[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:48:55 dist-test-slave-98hb krb5kdc[13911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:48:55 dist-test-slave-98hb krb5kdc[13911](info): set up 2 sockets
Sep 26 22:48:55 dist-test-slave-98hb krb5kdc[13911](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 22:48:56.365564 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:48:56 dist-test-slave-98hb krb5kdc[13911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:48:56.474844 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:38641
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:38641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:48:56.552312 13924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:48:56.552502 13924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:48:56.552522 13924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:48:56.553851 13924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:48:56.553903 13924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:48:56.553916 13924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:48:56.553928 13924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:48:56.553941 13924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:48:56.555414 13924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:38641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:38641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:48:56.555603 13924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:48:56.555771 13924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:48:56.560312 13930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:56.560276 13929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:56.560281 13932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:48:56.560298 13924 server_base.cc:1034] running on GCE node
I20260926 22:48:56.560689 13924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:48:56.560967 13924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:48:56.562098 13924 hybrid_clock.cc:648] HybridClock initialized: now 1790462936562086 us; error 30 us; skew 500 ppm
Sep 26 22:48:56 dist-test-slave-98hb krb5kdc[13911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:48:56.564292 13924 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:48:56.564670 13924 webserver.cc:460] Webserver started at http://127.13.148.62:39547/ using document root <none> and password file <none>
I20260926 22:48:56.564879 13924 fs_manager.cc:359] Metadata directory not provided
I20260926 22:48:56.564921 13924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:48:56.565032 13924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:48:56.566035 13924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "cbf28fba80c64b0f9defc50ff03136e1"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "eaa28a96b62008673e4db32abfec29f8"
server_key_iv: "b370a37d8de575674541054395dc51c0"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.566349 13924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "cbf28fba80c64b0f9defc50ff03136e1"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "eaa28a96b62008673e4db32abfec29f8"
server_key_iv: "b370a37d8de575674541054395dc51c0"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.567484 13924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:48:56.568126 13939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:56.568250 13924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:48:56.568342 13924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "cbf28fba80c64b0f9defc50ff03136e1"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "eaa28a96b62008673e4db32abfec29f8"
server_key_iv: "b370a37d8de575674541054395dc51c0"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.568396 13924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:48:56.576958 13924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:48:56.578543 13924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:48:56.578663 13924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:48:56.583002 13924 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:38641
I20260926 22:48:56.583032 13991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:38641 every 8 connection(s)
I20260926 22:48:56.583492 13924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:48:56.584152 13992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:48:56.586424 13992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1: Bootstrap starting.
I20260926 22:48:56.586969 13992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1: Neither blocks nor log segments found. Creating new log.
I20260926 22:48:56.587220 13992 log.cc:824] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1: Log is configured to *not* fsync() on all Append() calls
I20260926 22:48:56.587925 13992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1: No bootstrap required, opened a new log
I20260926 22:48:56.588981 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 13924
I20260926 22:48:56.589092 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:48:56.589901 13904 external_mini_cluster.cc:1428] Setting key c088a0bc9c0a224d1467990095c603d2
I20260926 22:48:56.589978 13992 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf28fba80c64b0f9defc50ff03136e1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38641 } }
I20260926 22:48:56.590144 13992 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:48:56.590185 13992 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf28fba80c64b0f9defc50ff03136e1, State: Initialized, Role: FOLLOWER
I20260926 22:48:56.590324 13992 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf28fba80c64b0f9defc50ff03136e1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38641 } }
I20260926 22:48:56.590399 13992 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:48:56.590433 13992 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:48:56.590479 13992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:48:56.591121 13992 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf28fba80c64b0f9defc50ff03136e1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38641 } }
I20260926 22:48:56.591215 13992 leader_election.cc:302] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf28fba80c64b0f9defc50ff03136e1; no voters:
I20260926 22:48:56.591418 13992 leader_election.cc:288] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:48:56.591429 13997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:48:56.591549 13997 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [term 1 LEADER]: Becoming Leader. State: Replica: cbf28fba80c64b0f9defc50ff03136e1, State: Running, Role: LEADER
I20260926 22:48:56.591676 13992 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:48:56.591675 13997 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf28fba80c64b0f9defc50ff03136e1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38641 } }
I20260926 22:48:56.592058 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf28fba80c64b0f9defc50ff03136e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf28fba80c64b0f9defc50ff03136e1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38641 } } }
I20260926 22:48:56.592073 13998 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf28fba80c64b0f9defc50ff03136e1. Latest consensus state: current_term: 1 leader_uuid: "cbf28fba80c64b0f9defc50ff03136e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf28fba80c64b0f9defc50ff03136e1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38641 } } }
I20260926 22:48:56.592181 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [sys.catalog]: This master's current role is: LEADER
I20260926 22:48:56.592213 13998 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1 [sys.catalog]: This master's current role is: LEADER
W20260926 22:48:56.593398 14013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:48:56.593452 14013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:48:56.593495 14005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:48:56.593644 14005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:48:56.594993 14005 catalog_manager.cc:1347] Generated new cluster ID: 41c1a847aa4141cca4e73ed68f7b4e0c
I20260926 22:48:56.595043 14005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:48:56 dist-test-slave-98hb krb5kdc[13911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:48:56.603801 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:56.590759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.13.148.62:38641)
0926 22:48:56.591317 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:56.591325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:56.591360 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:48:56.593454 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:56.593460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:56.593516 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:56.593639 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:56.595373 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.596243 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.597203 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.597611 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.597701 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:56.600681 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:56.600684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:56.600684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:56.600693 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:56.602197 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:56.602555 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:56.602558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:56.602559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:56.602646 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:56.602862 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:56.602863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:56.602863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:56.603075 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:56.603232 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:56.603476 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 22:48:56.603596 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":466,"threads_started":1}
I20260926 22:48:56.611490 14005 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:48:56.611940 14005 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:48:56.617200 14005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf28fba80c64b0f9defc50ff03136e1: Generated new TSK 0
I20260926 22:48:56.617345 14005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:48:56.638592 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38641
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:48:56.716892 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:48:56.717082 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:48:56.717121 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:48:56.718318 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:48:56.718358 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:48:56.718394 14018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:48:56.719830 14018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38641
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:48:56.720052 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:48:56.720266 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:48:56.724850 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:56.724843 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:56.724843 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:48:56.724931 14018 server_base.cc:1034] running on GCE node
I20260926 22:48:56.725229 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:48:56.725447 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:48:56.726609 14018 hybrid_clock.cc:648] HybridClock initialized: now 1790462936726596 us; error 29 us; skew 500 ppm
Sep 26 22:48:56 dist-test-slave-98hb krb5kdc[13911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:48:56.728858 14018 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:48:56.729245 14018 webserver.cc:460] Webserver started at http://127.13.148.1:41583/ using document root <none> and password file <none>
I20260926 22:48:56.729444 14018 fs_manager.cc:359] Metadata directory not provided
I20260926 22:48:56.729489 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:48:56.729610 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:48:56.730639 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "1dafc33790f84d689ea44199f870802d"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "0d633bc6677a32d0e2db19e46a7fa584"
server_key_iv: "7f62c8b5eea97d98be2088e39879bc16"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.730957 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "1dafc33790f84d689ea44199f870802d"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "0d633bc6677a32d0e2db19e46a7fa584"
server_key_iv: "7f62c8b5eea97d98be2088e39879bc16"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.732110 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:48:56.732764 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:56.732893 14018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:48:56.732969 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "1dafc33790f84d689ea44199f870802d"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "0d633bc6677a32d0e2db19e46a7fa584"
server_key_iv: "7f62c8b5eea97d98be2088e39879bc16"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.733032 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:48:56.751850 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:48:56.753875 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:48:56.754014 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:48:56.754240 14018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:48:56.754539 14018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:48:56.754573 14018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:56.754603 14018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:48:56.754619 14018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:56.760087 14018 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:44659
I20260926 22:48:56.760146 14146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:44659 every 8 connection(s)
I20260926 22:48:56.760432 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:48:56.762996 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 14018
I20260926 22:48:56.763093 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:48:56.763232 13904 external_mini_cluster.cc:1428] Setting key 274911ec4d5018fac8f133ce40558fae
Sep 26 22:48:56 dist-test-slave-98hb krb5kdc[13911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:48:56.769641 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:56.761155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:43787 (local address 127.13.148.62:38641)
0926 22:48:56.761208 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:56.761210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:56.761403 (+ 193us) server_negotiation.cc:407] Connection header received
0926 22:48:56.761889 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:56.761890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:56.761901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:56.761953 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:56.762896 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.763588 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.764573 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.764786 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.764841 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:56.767052 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:56.767054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:56.767054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:56.767062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:56.768328 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:56.768773 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:56.768774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:56.768774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:56.768817 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:56.769063 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:56.769063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:56.769064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:56.769155 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:56.769210 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:56.769513 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 22:48:56.769578 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:48:56.769992 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:56.761082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38641 (local address 127.13.148.1:43787)
0926 22:48:56.761269 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:56.761306 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:56.761809 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:56.762032 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:56.762036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:56.762592 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:48:56.762804 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:56.762809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.763672 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.763673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:56.764493 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:56.764496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.764855 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.764856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:56.764974 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:56.765448 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:48:56.765456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:48:56.766936 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:48:56.768404 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:56.768411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:56.768413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:56.768708 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:56.768870 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:56.768871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:56.768871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:56.769004 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:56.769217 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:48:56.769222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:48:56.769391 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 22:48:56.769505 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 22:48:56.769672 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260926 22:48:56.770524 14147 heartbeater.cc:347] Connected to a master server at 127.13.148.62:38641
I20260926 22:48:56.770613 14147 heartbeater.cc:464] Registering TS with master...
I20260926 22:48:56.770821 14147 heartbeater.cc:511] Master 127.13.148.62:38641 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260926 22:48:56.771333 13956 ts_manager.cc:194] Registered new tserver with Master: 1dafc33790f84d689ea44199f870802d (127.13.148.1:44659)
I20260926 22:48:56.772001 13956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:43787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:48:56.787703 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38641
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:48:56.866798 14152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:48:56.866993 14152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:48:56.867023 14152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:48:56.868129 14152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:48:56.868170 14152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:48:56.868212 14152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:48:56.869627 14152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38641
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:48:56.869880 14152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:48:56.870085 14152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:48:56.874876 14152 server_base.cc:1034] running on GCE node
W20260926 22:48:56.874835 14158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:56.874841 14157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:56.874820 14160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:48:56.875253 14152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:48:56.875475 14152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:48:56.876619 14152 hybrid_clock.cc:648] HybridClock initialized: now 1790462936876604 us; error 27 us; skew 500 ppm
Sep 26 22:48:56 dist-test-slave-98hb krb5kdc[13911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:48:56.878963 14152 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:48:56.879348 14152 webserver.cc:460] Webserver started at http://127.13.148.2:45421/ using document root <none> and password file <none>
I20260926 22:48:56.879547 14152 fs_manager.cc:359] Metadata directory not provided
I20260926 22:48:56.879593 14152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:48:56.879712 14152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:48:56.880681 14152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "dbe927631edd4372a2058cf9a6db7b31"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "43b2058f6e329e1857816d72fd72b607"
server_key_iv: "eebd2f838c1082509433c81143730220"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.880995 14152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "dbe927631edd4372a2058cf9a6db7b31"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "43b2058f6e329e1857816d72fd72b607"
server_key_iv: "eebd2f838c1082509433c81143730220"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.882241 14152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:48:56.882854 14167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:56.882994 14152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:48:56.883068 14152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "dbe927631edd4372a2058cf9a6db7b31"
format_stamp: "Formatted at 2026-09-26 22:48:56 on dist-test-slave-98hb"
server_key: "43b2058f6e329e1857816d72fd72b607"
server_key_iv: "eebd2f838c1082509433c81143730220"
server_key_version: "encryptionkey@0"
I20260926 22:48:56.883126 14152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:48:56.915594 14152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:48:56.917514 14152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:48:56.917652 14152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:48:56.917899 14152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:48:56.918185 14152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:48:56.918218 14152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:56.918246 14152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:48:56.918264 14152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:56.923661 14152 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:42503
I20260926 22:48:56.923731 14280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:42503 every 8 connection(s)
I20260926 22:48:56.924007 14152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:48:56 dist-test-slave-98hb krb5kdc[13911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462936, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:48:56.931905 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:56.924665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39013 (local address 127.13.148.62:38641)
0926 22:48:56.924752 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:56.924754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:56.924885 (+ 131us) server_negotiation.cc:407] Connection header received
0926 22:48:56.925346 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:56.925347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:56.925356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:56.925402 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:56.926284 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.926777 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.927675 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.927861 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.927904 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:56.929806 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:56.929807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:56.929808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:56.929814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:56.930893 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:56.931212 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:56.931213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:56.931213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:56.931257 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:56.931436 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:56.931436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:56.931436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:56.931532 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:56.931579 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:56.931779 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 22:48:56.931840 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 22:48:56.932085 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:56.924640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38641 (local address 127.13.148.2:39013)
0926 22:48:56.924774 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:56.924809 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:56.925272 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:56.925458 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:56.925461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:56.926019 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:48:56.926215 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:56.926220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.926839 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.926839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:56.927616 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:56.927618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:56.927914 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:56.927915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:56.928008 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:56.928468 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:48:56.928476 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:48:56.929694 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:48:56.930954 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:56.930958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:56.930960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:56.931158 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:56.931304 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:56.931304 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:56.931304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:56.931392 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:56.931576 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:48:56.931578 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:48:56.931684 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:48:56.931764 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 22:48:56.931878 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260926 22:48:56.932297 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 14152
I20260926 22:48:56.932392 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:48:56.932560 13904 external_mini_cluster.cc:1428] Setting key 69982fa54418b4327dab4758d7589c2d
I20260926 22:48:56.932585 14281 heartbeater.cc:347] Connected to a master server at 127.13.148.62:38641
I20260926 22:48:56.932680 14281 heartbeater.cc:464] Registering TS with master...
I20260926 22:48:56.932873 14281 heartbeater.cc:511] Master 127.13.148.62:38641 requested a full tablet report, sending...
I20260926 22:48:56.933219 13956 ts_manager.cc:194] Registered new tserver with Master: dbe927631edd4372a2058cf9a6db7b31 (127.13.148.2:42503)
I20260926 22:48:56.933564 13956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:39013
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:48:56.954854 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38641
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:48:57.032897 14286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:48:57.033095 14286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:48:57.033118 14286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:48:57.034253 14286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:48:57.034293 14286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:48:57.034318 14286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:48:57.035772 14286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38641
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:48:57.035987 14286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:48:57.036226 14286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:48:57.040926 14294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:57.040930 14292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:57.040959 14291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:48:57.041067 14286 server_base.cc:1034] running on GCE node
I20260926 22:48:57.041365 14286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:48:57.041584 14286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:48:57.042711 14286 hybrid_clock.cc:648] HybridClock initialized: now 1790462937042692 us; error 37 us; skew 500 ppm
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[13911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:48:57.045107 14286 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:48:57.045507 14286 webserver.cc:460] Webserver started at http://127.13.148.3:36391/ using document root <none> and password file <none>
I20260926 22:48:57.045679 14286 fs_manager.cc:359] Metadata directory not provided
I20260926 22:48:57.045717 14286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:48:57.045830 14286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:48:57.046790 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "e25feeed8a9a432ba7a26b3f3b06bb1e"
format_stamp: "Formatted at 2026-09-26 22:48:57 on dist-test-slave-98hb"
server_key: "6f2a1660364a70e122c497baf144fdd5"
server_key_iv: "e11deda6ea90aa4492f1a2268f9291b8"
server_key_version: "encryptionkey@0"
I20260926 22:48:57.047075 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "e25feeed8a9a432ba7a26b3f3b06bb1e"
format_stamp: "Formatted at 2026-09-26 22:48:57 on dist-test-slave-98hb"
server_key: "6f2a1660364a70e122c497baf144fdd5"
server_key_iv: "e11deda6ea90aa4492f1a2268f9291b8"
server_key_version: "encryptionkey@0"
I20260926 22:48:57.048214 14286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:48:57.048870 14301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:57.048995 14286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:48:57.049070 14286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "e25feeed8a9a432ba7a26b3f3b06bb1e"
format_stamp: "Formatted at 2026-09-26 22:48:57 on dist-test-slave-98hb"
server_key: "6f2a1660364a70e122c497baf144fdd5"
server_key_iv: "e11deda6ea90aa4492f1a2268f9291b8"
server_key_version: "encryptionkey@0"
I20260926 22:48:57.049139 14286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:48:57.072849 14286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:48:57.074667 14286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:48:57.074790 14286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:48:57.075001 14286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:48:57.075313 14286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:48:57.075346 14286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:57.075367 14286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:48:57.075390 14286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:57.080964 14286 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36099
I20260926 22:48:57.081012 14414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36099 every 8 connection(s)
I20260926 22:48:57.081302 14286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[13911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462937, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:48:57.089172 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 14286
I20260926 22:48:57.089272 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:48:57.089449 13904 external_mini_cluster.cc:1428] Setting key 45003c4a1c605acb08eebd90db6ed7ff
I20260926 22:48:57.089612 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:57.081942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:41005 (local address 127.13.148.62:38641)
0926 22:48:57.082017 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:57.082019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:57.082181 (+ 162us) server_negotiation.cc:407] Connection header received
0926 22:48:57.082669 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:57.082670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:57.082681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:57.082732 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:57.083634 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:57.084127 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:57.085060 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:57.085295 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:57.085347 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:57.087334 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:57.087335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:57.087335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:57.087341 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:57.088462 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:57.088801 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:57.088802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:57.088802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:57.088846 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:57.089031 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:57.089031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:57.089031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:57.089120 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:57.089162 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:57.089463 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 22:48:57.089531 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 22:48:57.089856 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:57.081916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38641 (local address 127.13.148.3:41005)
0926 22:48:57.082074 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:57.082111 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:57.082582 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:57.082801 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:57.082805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:57.083359 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:48:57.083546 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:57.083551 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:57.084203 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:57.084203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:57.084982 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:57.084984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:57.085361 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:57.085362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:57.085488 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:57.085936 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:48:57.085943 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:48:57.087248 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:48:57.088523 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:57.088528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:57.088530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:57.088745 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:57.088898 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:57.088898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:57.088898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:57.088984 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:57.089170 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:48:57.089175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:48:57.089335 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 22:48:57.089444 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 22:48:57.089605 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260926 22:48:57.090247 14415 heartbeater.cc:347] Connected to a master server at 127.13.148.62:38641
I20260926 22:48:57.090397 14415 heartbeater.cc:464] Registering TS with master...
I20260926 22:48:57.090569 14415 heartbeater.cc:511] Master 127.13.148.62:38641 requested a full tablet report, sending...
I20260926 22:48:57.090893 13956 ts_manager.cc:194] Registered new tserver with Master: e25feeed8a9a432ba7a26b3f3b06bb1e (127.13.148.3:36099)
I20260926 22:48:57.091257 13956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:41005
I20260926 22:48:57.101325 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[13911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[13911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:48:57.122750 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:57.115683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.13.148.1:44659)
0926 22:48:57.115836 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:57.115841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:57.115851 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:48:57.115879 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:57.115881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:57.115894 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:57.115955 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:57.116504 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:57.117213 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:57.117957 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:57.118286 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:57.118347 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:57.120368 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:57.120377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:57.120377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:57.120384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:57.121657 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:57.122020 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:57.122022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:57.122023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:57.122074 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:57.122237 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:57.122238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:57.122239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:57.122366 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:57.122529 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:57.122570 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 22:48:57.122643 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
W20260926 22:48:57.123083 14061 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60916
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[13911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[13911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:48:57.142967 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:57.136855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.13.148.1:44659)
0926 22:48:57.136918 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:57.136920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:57.136932 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:48:57.137005 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:57.137006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:57.137017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:57.137061 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:57.137582 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:57.138108 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:57.138794 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:57.138991 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:57.139043 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:57.140970 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:57.140972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:57.140972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:57.140979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:57.142045 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:57.142333 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:57.142334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:57.142334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:57.142381 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:57.142535 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:57.142535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:57.142536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:57.142622 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:57.142717 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:57.142805 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 22:48:57.142877 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:48:57.143430 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 14018
I20260926 22:48:57.147672 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 14152
I20260926 22:48:57.151325 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 14286
I20260926 22:48:57.154683 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 13924
2026-09-26T22:48:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1746 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[14429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[14429](info): set up 2 sockets
Sep 26 22:48:57 dist-test-slave-98hb krb5kdc[14429](info): commencing operation
krb5kdc: starting...
W20260926 22:48:58.105345 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:48:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:48:58.223971 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35483
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:48:58.302279 14442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:48:58.302480 14442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:48:58.302510 14442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:48:58.303633 14442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:48:58.303674 14442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:48:58.303692 14442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:48:58.303709 14442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:48:58.303723 14442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:48:58.305182 14442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:48:58.305424 14442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:48:58.305639 14442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:48:58.310349 14447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:58.310355 14450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:58.310355 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:48:58.311072 14442 server_base.cc:1034] running on GCE node
I20260926 22:48:58.311295 14442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:48:58.311544 14442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:48:58.312693 14442 hybrid_clock.cc:648] HybridClock initialized: now 1790462938312683 us; error 26 us; skew 500 ppm
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:48:58.315095 14442 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:48:58.315531 14442 webserver.cc:460] Webserver started at http://127.13.148.62:40465/ using document root <none> and password file <none>
I20260926 22:48:58.315750 14442 fs_manager.cc:359] Metadata directory not provided
I20260926 22:48:58.315794 14442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:48:58.315910 14442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:48:58.316980 14442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "fd9d132131de4997b6e4a5680ad599a0"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "f8d23aae1276bd4bd6bafd7ec4565db7"
server_key_iv: "c13c01afc9720bcabc6bfe3e43c0966c"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.317293 14442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "fd9d132131de4997b6e4a5680ad599a0"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "f8d23aae1276bd4bd6bafd7ec4565db7"
server_key_iv: "c13c01afc9720bcabc6bfe3e43c0966c"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.318572 14442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 22:48:58.319248 14457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.319406 14442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:48:58.319476 14442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "fd9d132131de4997b6e4a5680ad599a0"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "f8d23aae1276bd4bd6bafd7ec4565db7"
server_key_iv: "c13c01afc9720bcabc6bfe3e43c0966c"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.319541 14442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:48:58.335605 14442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:48:58.337405 14442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:48:58.337530 14442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:48:58.341223 14442 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35483
I20260926 22:48:58.341257 14509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35483 every 8 connection(s)
I20260926 22:48:58.341569 14442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:48:58.342386 14510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:48:58.344612 14510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0: Bootstrap starting.
I20260926 22:48:58.345141 14510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0: Neither blocks nor log segments found. Creating new log.
I20260926 22:48:58.345395 14510 log.cc:824] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0: Log is configured to *not* fsync() on all Append() calls
I20260926 22:48:58.346052 14510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0: No bootstrap required, opened a new log
I20260926 22:48:58.347571 14510 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9d132131de4997b6e4a5680ad599a0" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35483 } }
I20260926 22:48:58.347705 14510 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:48:58.347735 14510 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9d132131de4997b6e4a5680ad599a0, State: Initialized, Role: FOLLOWER
I20260926 22:48:58.347863 14510 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9d132131de4997b6e4a5680ad599a0" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35483 } }
I20260926 22:48:58.347923 14510 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:48:58.347951 14510 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:48:58.347997 14510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:48:58.348482 14510 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9d132131de4997b6e4a5680ad599a0" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35483 } }
I20260926 22:48:58.348572 14510 leader_election.cc:302] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd9d132131de4997b6e4a5680ad599a0; no voters:
I20260926 22:48:58.348727 14510 leader_election.cc:288] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:48:58.348727 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 14442
I20260926 22:48:58.348762 14513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:48:58.348796 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:48:58.348881 14513 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [term 1 LEADER]: Becoming Leader. State: Replica: fd9d132131de4997b6e4a5680ad599a0, State: Running, Role: LEADER
I20260926 22:48:58.348944 13904 external_mini_cluster.cc:1428] Setting key d2f81084385c9761fc90d754ee7c779d
I20260926 22:48:58.348973 14510 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:48:58.348955 14513 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9d132131de4997b6e4a5680ad599a0" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35483 } }
I20260926 22:48:58.349421 14515 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd9d132131de4997b6e4a5680ad599a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9d132131de4997b6e4a5680ad599a0" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35483 } } }
I20260926 22:48:58.349547 14515 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [sys.catalog]: This master's current role is: LEADER
I20260926 22:48:58.349750 14514 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd9d132131de4997b6e4a5680ad599a0. Latest consensus state: current_term: 1 leader_uuid: "fd9d132131de4997b6e4a5680ad599a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9d132131de4997b6e4a5680ad599a0" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35483 } } }
I20260926 22:48:58.349790 14524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:48:58.350435 14524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:48:58.349819 14514 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0 [sys.catalog]: This master's current role is: LEADER
I20260926 22:48:58.352697 14524 catalog_manager.cc:1347] Generated new cluster ID: cca63c2314bb464496ce4fb1873c5395
I20260926 22:48:58.352751 14524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:48:58.360270 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.349406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.13.148.62:35483)
0926 22:48:58.349953 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.349963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.349994 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:48:58.350111 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.350119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.350191 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.350335 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:58.351094 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.352213 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.353197 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.353869 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.354008 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.356680 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:58.356685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:58.356685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:58.356698 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:58.358547 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.358976 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.358978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.358979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.359093 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.359287 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.359288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.359288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.359593 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:58.359725 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.359873 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.360010 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":57,"threads_started":1}
I20260926 22:48:58.368456 14524 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:48:58.369213 14524 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:48:58.376106 14524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd9d132131de4997b6e4a5680ad599a0: Generated new TSK 0
I20260926 22:48:58.376324 14524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:48:58.400223 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35483
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:48:58.478533 14536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:48:58.478730 14536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:48:58.478753 14536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:48:58.479892 14536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:48:58.479930 14536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:48:58.479960 14536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:48:58.481416 14536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:48:58.481638 14536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:48:58.481863 14536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:48:58.486461 14542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:58.486435 14541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:58.486435 14544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:48:58.486526 14536 server_base.cc:1034] running on GCE node
I20260926 22:48:58.486857 14536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:48:58.487079 14536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:48:58.488204 14536 hybrid_clock.cc:648] HybridClock initialized: now 1790462938488185 us; error 31 us; skew 500 ppm
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:48:58.490451 14536 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:48:58.490824 14536 webserver.cc:460] Webserver started at http://127.13.148.1:35693/ using document root <none> and password file <none>
I20260926 22:48:58.491020 14536 fs_manager.cc:359] Metadata directory not provided
I20260926 22:48:58.491065 14536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:48:58.491184 14536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:48:58.492130 14536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "572872fb23674259a8503fb35a977b3d"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "82066e1d6ba42d21ba6a94f212458a7a"
server_key_iv: "1fbc4a2edafde463a969353e2a51f82b"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.492434 14536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "572872fb23674259a8503fb35a977b3d"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "82066e1d6ba42d21ba6a94f212458a7a"
server_key_iv: "1fbc4a2edafde463a969353e2a51f82b"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.493582 14536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:48:58.494392 14551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.494544 14536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:48:58.494622 14536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "572872fb23674259a8503fb35a977b3d"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "82066e1d6ba42d21ba6a94f212458a7a"
server_key_iv: "1fbc4a2edafde463a969353e2a51f82b"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.494679 14536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:48:58.512226 14536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:48:58.513911 14536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:48:58.514037 14536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:48:58.514237 14536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:48:58.514544 14536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:48:58.514575 14536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.514601 14536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:48:58.514626 14536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.519906 14536 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:33681
I20260926 22:48:58.519982 14664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:33681 every 8 connection(s)
I20260926 22:48:58.520227 14536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:48:58.524592 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 14536
I20260926 22:48:58.524674 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:48:58.524820 13904 external_mini_cluster.cc:1428] Setting key a82c4437418e070b9040bed8386fa050
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:48:58.529311 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.520872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:46613 (local address 127.13.148.62:35483)
0926 22:48:58.520922 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.520924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.521105 (+ 181us) server_negotiation.cc:407] Connection header received
0926 22:48:58.522039 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.522040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.522055 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.522116 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:58.522690 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.523252 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.524174 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.524366 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.524407 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.526660 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:58.526661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:58.526662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:58.526668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:58.528035 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.528389 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.528390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.528390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.528451 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.528653 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.528654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.528654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.528788 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:58.528855 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.529110 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.529223 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 22:48:58.529341 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.520845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35483 (local address 127.13.148.1:46613)
0926 22:48:58.520993 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:58.521033 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:58.521938 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:58.522184 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:58.522188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:58.522447 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:48:58.522625 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.522629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.523321 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.523322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.524114 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.524116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.524415 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.524416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.524509 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.524990 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:48:58.525000 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:48:58.526574 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:48:58.528104 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:58.528110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:58.528112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:58.528323 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:58.528505 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:58.528506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:58.528506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:58.528598 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:58.528844 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:48:58.528847 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:48:58.528991 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 22:48:58.529072 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 22:48:58.529185 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260926 22:48:58.529764 14665 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35483
I20260926 22:48:58.529861 14665 heartbeater.cc:464] Registering TS with master...
I20260926 22:48:58.530041 14665 heartbeater.cc:511] Master 127.13.148.62:35483 requested a full tablet report, sending...
I20260926 22:48:58.530542 14474 ts_manager.cc:194] Registered new tserver with Master: 572872fb23674259a8503fb35a977b3d (127.13.148.1:33681)
I20260926 22:48:58.531129 14474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:46613
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:48:58.548767 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35483
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:48:58.627241 14670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:48:58.627449 14670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:48:58.627471 14670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:48:58.628604 14670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:48:58.628643 14670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:48:58.628667 14670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:48:58.630111 14670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:48:58.630328 14670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:48:58.630535 14670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:48:58.635181 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:58.635222 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:58.635177 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:48:58.635357 14670 server_base.cc:1034] running on GCE node
I20260926 22:48:58.635542 14670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:48:58.635761 14670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:48:58.636898 14670 hybrid_clock.cc:648] HybridClock initialized: now 1790462938636884 us; error 30 us; skew 500 ppm
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:48:58.639115 14670 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:48:58.639496 14670 webserver.cc:460] Webserver started at http://127.13.148.2:45361/ using document root <none> and password file <none>
I20260926 22:48:58.639678 14670 fs_manager.cc:359] Metadata directory not provided
I20260926 22:48:58.639724 14670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:48:58.639849 14670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:48:58.640846 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "09e1ca7742db4ccbbce34d5d7e86e66a"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "9d715a8e4a019f58d6ff610d6c3e5229"
server_key_iv: "cb3136cae0625aca896f81ded833b6fa"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.641157 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "09e1ca7742db4ccbbce34d5d7e86e66a"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "9d715a8e4a019f58d6ff610d6c3e5229"
server_key_iv: "cb3136cae0625aca896f81ded833b6fa"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.642381 14670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:48:58.643038 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.643163 14670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:48:58.643239 14670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "09e1ca7742db4ccbbce34d5d7e86e66a"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "9d715a8e4a019f58d6ff610d6c3e5229"
server_key_iv: "cb3136cae0625aca896f81ded833b6fa"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.643294 14670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:48:58.671788 14670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:48:58.673497 14670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:48:58.673630 14670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:48:58.673866 14670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:48:58.674149 14670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:48:58.674180 14670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.674209 14670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:48:58.674232 14670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.679390 14670 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:37029
I20260926 22:48:58.679454 14798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:37029 every 8 connection(s)
I20260926 22:48:58.679718 14670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:48:58.683070 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 14670
I20260926 22:48:58.683167 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:48:58.683329 13904 external_mini_cluster.cc:1428] Setting key b75b70a4602bb572fcd54b2746147803
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:48:58.688970 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.680381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:40111 (local address 127.13.148.62:35483)
0926 22:48:58.680441 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.680442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.680624 (+ 182us) server_negotiation.cc:407] Connection header received
0926 22:48:58.681547 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.681549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.681561 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.681611 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:58.682233 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.682741 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.684062 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.684251 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.684303 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.686592 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:58.686594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:58.686594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:58.686600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:58.687814 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.688160 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.688161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.688162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.688213 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.688412 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.688413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.688413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.688516 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:58.688569 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.688805 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.688883 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:48:58.689138 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.680357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35483 (local address 127.13.148.2:40111)
0926 22:48:58.680512 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:58.680548 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:58.681441 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:58.681683 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:58.681687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:58.681953 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:48:58.682142 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.682146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.682820 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.682822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.683987 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.683990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.684324 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.684325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.684462 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.684905 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:48:58.684915 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:48:58.686512 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:48:58.687886 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:58.687892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:58.687894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:58.688099 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:58.688264 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:58.688265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:58.688265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:58.688361 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:58.688566 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:48:58.688572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:48:58.688698 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 22:48:58.688781 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:48:58.688932 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260926 22:48:58.689520 14799 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35483
I20260926 22:48:58.689617 14799 heartbeater.cc:464] Registering TS with master...
I20260926 22:48:58.689828 14799 heartbeater.cc:511] Master 127.13.148.62:35483 requested a full tablet report, sending...
I20260926 22:48:58.690157 14474 ts_manager.cc:194] Registered new tserver with Master: 09e1ca7742db4ccbbce34d5d7e86e66a (127.13.148.2:37029)
I20260926 22:48:58.690515 14474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:40111
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:48:58.707645 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35483
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:48:58.785760 14804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:48:58.785948 14804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:48:58.785970 14804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:48:58.787051 14804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:48:58.787087 14804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:48:58.787111 14804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:48:58.788528 14804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:48:58.788710 14804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:48:58.788877 14804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:48:58.793468 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:58.793442 14812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:48:58.793505 14810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:48:58.793678 14804 server_base.cc:1034] running on GCE node
I20260926 22:48:58.793910 14804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:48:58.794123 14804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:48:58.795255 14804 hybrid_clock.cc:648] HybridClock initialized: now 1790462938795242 us; error 27 us; skew 500 ppm
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:48:58.797605 14804 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:48:58.798008 14804 webserver.cc:460] Webserver started at http://127.13.148.3:36405/ using document root <none> and password file <none>
I20260926 22:48:58.798197 14804 fs_manager.cc:359] Metadata directory not provided
I20260926 22:48:58.798237 14804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:48:58.798372 14804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:48:58.799342 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "339d79537ddf4f259e28b6b22e259932"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "50ce09e01d6e2ec7efa9c972148cec1a"
server_key_iv: "6836152bd3b3d66c3b669b1f8b1c978c"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.799644 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "339d79537ddf4f259e28b6b22e259932"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "50ce09e01d6e2ec7efa9c972148cec1a"
server_key_iv: "6836152bd3b3d66c3b669b1f8b1c978c"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.800787 14804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:48:58.801422 14819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.801561 14804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:48:58.801630 14804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "339d79537ddf4f259e28b6b22e259932"
format_stamp: "Formatted at 2026-09-26 22:48:58 on dist-test-slave-98hb"
server_key: "50ce09e01d6e2ec7efa9c972148cec1a"
server_key_iv: "6836152bd3b3d66c3b669b1f8b1c978c"
server_key_version: "encryptionkey@0"
I20260926 22:48:58.801692 14804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:48:58.828890 14804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:48:58.830955 14804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:48:58.831075 14804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:48:58.831287 14804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:48:58.831583 14804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:48:58.831614 14804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.831634 14804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:48:58.831656 14804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:48:58.836987 14804 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:33667
I20260926 22:48:58.837049 14932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:33667 every 8 connection(s)
I20260926 22:48:58.837320 14804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:48:58.841936 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 14804
I20260926 22:48:58.842041 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:48:58.842216 13904 external_mini_cluster.cc:1428] Setting key 7ae423ca374404edc583e3583ea6c630
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:48:58.845829 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.837987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:46949 (local address 127.13.148.62:35483)
0926 22:48:58.838040 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.838042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.838248 (+ 206us) server_negotiation.cc:407] Connection header received
0926 22:48:58.839189 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.839190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.839200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.839251 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:58.839844 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.840352 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.841307 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.841481 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.841524 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.843628 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:58.843630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:58.843630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:58.843636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:58.844791 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.845084 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.845085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.845085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.845129 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.845320 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.845320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.845321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.845411 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:58.845452 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.845650 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.845755 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:48:58.846016 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.837970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35483 (local address 127.13.148.3:46949)
0926 22:48:58.838124 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:58.838165 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:58.839093 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:58.839316 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:58.839319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:58.839560 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:48:58.839771 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.839775 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.840433 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.840434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.841238 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.841240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.841535 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.841536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.841630 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.842065 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:48:58.842076 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:48:58.843547 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:48:58.844848 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:58.844851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:58.844853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:58.845035 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:58.845169 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:48:58.845170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:48:58.845170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:48:58.845269 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:48:58.845452 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:48:58.845456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:48:58.845562 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:48:58.845637 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 22:48:58.845801 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260926 22:48:58.846411 14933 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35483
I20260926 22:48:58.846493 14933 heartbeater.cc:464] Registering TS with master...
I20260926 22:48:58.846652 14933 heartbeater.cc:511] Master 127.13.148.62:35483 requested a full tablet report, sending...
I20260926 22:48:58.846975 14474 ts_manager.cc:194] Registered new tserver with Master: 339d79537ddf4f259e28b6b22e259932 (127.13.148.3:33667)
I20260926 22:48:58.847321 14474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:46949
I20260926 22:48:58.853652 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:48:58.860157 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.855093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.13.148.62:35483)
0926 22:48:58.855142 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.855144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.855246 (+ 102us) server_negotiation.cc:407] Connection header received
0926 22:48:58.855300 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.855301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.855311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.855352 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:58.855875 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.856351 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.857015 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.857195 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.857247 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.858249 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:58.858250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:58.858251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:58.858257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:58.859197 (+ 940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.859490 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.859491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.859491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.859540 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.859691 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.859692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.859692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.859825 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:58.859907 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.860008 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.860073 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:48:58.861915 14474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:48:58.863348 14474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:48:58.868911 14942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.865135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:33667 (local address 127.0.0.1:59166)
0926 22:48:58.865280 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:58.865335 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:58.865396 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:58.865685 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:58.865689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:58.865691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:48:58.865774 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.865775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.867061 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.867062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.867881 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.867884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.868683 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.868684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.868750 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.868760 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 22:48:58.868812 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:48:58.868860 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":41,"thread_start_us":50,"threads_started":1}
I20260926 22:48:58.869114 14945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.865292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.13.148.3:33667)
0926 22:48:58.865520 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.865525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.865535 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:48:58.865572 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.865574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.865590 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.865674 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:48:58.865868 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.866976 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.867972 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.868615 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.868686 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.868739 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.868838 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.868905 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":59,"threads_started":1}
I20260926 22:48:58.869714 14867 tablet_service.cc:1467] Processing CreateTablet for tablet 68824e7ee00c4abc96fd1695c1147b7f (DEFAULT_TABLE table=test-table [id=f3630b099387496a86afb2c6269a9996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:48:58.869987 14944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.865135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37029 (local address 127.0.0.1:59018)
0926 22:48:58.865378 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:58.865386 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:58.865422 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:58.866057 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:58.866058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:58.866060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:48:58.866092 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.866093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.866957 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.866958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.868229 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.868231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.869755 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.869757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.869819 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.869823 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:48:58.869882 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:48:58.869923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":33,"thread_start_us":48,"threads_started":1}
I20260926 22:48:58.870019 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.865240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.13.148.2:37029)
0926 22:48:58.865823 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.865827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.865836 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:48:58.865874 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.865876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.865889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.865956 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:48:58.866164 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.866864 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.869141 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.869643 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.869699 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.869767 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.869903 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.869942 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":101,"threads_started":1}
I20260926 22:48:58.870087 14867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68824e7ee00c4abc96fd1695c1147b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:48:58.870136 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.865491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.13.148.1:33681)
0926 22:48:58.865649 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.865654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.865664 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:48:58.865696 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.865698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.865711 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.865813 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:48:58.865938 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.866967 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.868312 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.868985 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.869051 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.869104 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.870027 (+ 923us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.870064 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260926 22:48:58.870244 14941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.865308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:33681 (local address 127.0.0.1:56836)
0926 22:48:58.865476 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:58.865483 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:58.865512 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:58.865813 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:58.865814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:58.865816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:48:58.865869 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.865871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.867142 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.867143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.868228 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.868231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.869807 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.869808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.869874 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.869878 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:48:58.869929 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:48:58.869968 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":31,"threads_started":1}
I20260926 22:48:58.870906 14599 tablet_service.cc:1467] Processing CreateTablet for tablet 68824e7ee00c4abc96fd1695c1147b7f (DEFAULT_TABLE table=test-table [id=f3630b099387496a86afb2c6269a9996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:48:58.871141 14599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68824e7ee00c4abc96fd1695c1147b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:48:58.872596 14947 tablet_bootstrap.cc:492] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932: Bootstrap starting.
I20260926 22:48:58.872920 14733 tablet_service.cc:1467] Processing CreateTablet for tablet 68824e7ee00c4abc96fd1695c1147b7f (DEFAULT_TABLE table=test-table [id=f3630b099387496a86afb2c6269a9996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:48:58.873157 14733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68824e7ee00c4abc96fd1695c1147b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:48:58.873366 14947 tablet_bootstrap.cc:654] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932: Neither blocks nor log segments found. Creating new log.
I20260926 22:48:58.873677 14948 tablet_bootstrap.cc:492] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d: Bootstrap starting.
I20260926 22:48:58.873939 14947 log.cc:824] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932: Log is configured to *not* fsync() on all Append() calls
I20260926 22:48:58.874331 14948 tablet_bootstrap.cc:654] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d: Neither blocks nor log segments found. Creating new log.
I20260926 22:48:58.874853 14947 tablet_bootstrap.cc:492] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932: No bootstrap required, opened a new log
I20260926 22:48:58.874866 14948 log.cc:824] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:48:58.874912 14947 ts_tablet_manager.cc:1397] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:48:58.875564 14948 tablet_bootstrap.cc:492] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d: No bootstrap required, opened a new log
I20260926 22:48:58.875634 14948 ts_tablet_manager.cc:1397] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:48:58.876186 14951 tablet_bootstrap.cc:492] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a: Bootstrap starting.
I20260926 22:48:58.876507 14947 raft_consensus.cc:348] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.876682 14947 raft_consensus.cc:374] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:48:58.876720 14947 raft_consensus.cc:729] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 339d79537ddf4f259e28b6b22e259932, State: Initialized, Role: FOLLOWER
I20260926 22:48:58.876820 14947 consensus_queue.cc:260] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.876971 14951 tablet_bootstrap.cc:654] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a: Neither blocks nor log segments found. Creating new log.
I20260926 22:48:58.877094 14947 ts_tablet_manager.cc:1428] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 22:48:58.877223 14933 heartbeater.cc:503] Master 127.13.148.62:35483 was elected leader, sending a full tablet report...
I20260926 22:48:58.877432 14951 log.cc:824] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:48:58.877579 14948 raft_consensus.cc:348] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.877763 14948 raft_consensus.cc:374] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:48:58.877799 14948 raft_consensus.cc:729] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572872fb23674259a8503fb35a977b3d, State: Initialized, Role: FOLLOWER
I20260926 22:48:58.877934 14948 consensus_queue.cc:260] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.878021 14951 tablet_bootstrap.cc:492] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a: No bootstrap required, opened a new log
I20260926 22:48:58.878073 14951 ts_tablet_manager.cc:1397] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:48:58.878203 14948 ts_tablet_manager.cc:1428] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:48:58.878247 14665 heartbeater.cc:503] Master 127.13.148.62:35483 was elected leader, sending a full tablet report...
I20260926 22:48:58.879575 14951 raft_consensus.cc:348] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.879689 14951 raft_consensus.cc:374] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:48:58.879719 14951 raft_consensus.cc:729] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e1ca7742db4ccbbce34d5d7e86e66a, State: Initialized, Role: FOLLOWER
I20260926 22:48:58.879819 14951 consensus_queue.cc:260] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.880030 14951 ts_tablet_manager.cc:1428] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:48:58.880059 14799 heartbeater.cc:503] Master 127.13.148.62:35483 was elected leader, sending a full tablet report...
W20260926 22:48:58.930253 14800 tablet.cc:2369] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:48:58.955240 14952 raft_consensus.cc:482] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:48:58.955394 14952 raft_consensus.cc:504] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.955777 14952 leader_election.cc:288] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09e1ca7742db4ccbbce34d5d7e86e66a (127.13.148.2:37029), 572872fb23674259a8503fb35a977b3d (127.13.148.1:33681)
I20260926 22:48:58.959334 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.955994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:52521 (local address 127.13.148.2:37029)
0926 22:48:58.956035 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.956037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.956048 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:48:58.956074 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.956075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.956084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.956150 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:48:58.956315 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.957039 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.958440 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.958994 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.959064 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.959072 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.959226 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.959260 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:48:58.959326 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.955991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:39547 (local address 127.13.148.1:33681)
0926 22:48:58.956031 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.956032 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.956192 (+ 160us) server_negotiation.cc:407] Connection header received
0926 22:48:58.956231 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.956232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.956242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.956278 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:48:58.956410 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.957092 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.958436 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.958994 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.959064 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.959072 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.959226 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.959260 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:48:58.959349 14935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.955910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:37029 (local address 127.13.148.3:52521)
0926 22:48:58.955964 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:58.955975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:58.956014 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:58.956164 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:58.956165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:58.956167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:48:58.956239 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.956241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.957111 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.957111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.958357 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.958360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.959066 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.959067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.959171 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.959173 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:48:58.959219 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:48:58.959260 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":10}
I20260926 22:48:58.959362 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.955948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:33681 (local address 127.13.148.3:39547)
0926 22:48:58.956117 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 22:48:58.956126 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:48:58.956179 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:48:58.956292 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:48:58.956292 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:48:58.956294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:48:58.956352 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.956353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.957152 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.957152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.958359 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:48:58.958360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.959066 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.959067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:48:58.959159 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.959163 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:48:58.959223 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 22:48:58.959260 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260926 22:48:58.959664 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68824e7ee00c4abc96fd1695c1147b7f" candidate_uuid: "339d79537ddf4f259e28b6b22e259932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" is_pre_election: true
I20260926 22:48:58.959738 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68824e7ee00c4abc96fd1695c1147b7f" candidate_uuid: "339d79537ddf4f259e28b6b22e259932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572872fb23674259a8503fb35a977b3d" is_pre_election: true
I20260926 22:48:58.959820 14753 raft_consensus.cc:2454] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 339d79537ddf4f259e28b6b22e259932 in term 0.
I20260926 22:48:58.959892 14619 raft_consensus.cc:2454] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 339d79537ddf4f259e28b6b22e259932 in term 0.
I20260926 22:48:58.960031 14822 leader_election.cc:302] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09e1ca7742db4ccbbce34d5d7e86e66a, 339d79537ddf4f259e28b6b22e259932; no voters:
I20260926 22:48:58.960160 14952 raft_consensus.cc:2781] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:48:58.960224 14952 raft_consensus.cc:482] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:48:58.960247 14952 raft_consensus.cc:3037] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:48:58.960891 14952 raft_consensus.cc:504] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.961014 14952 leader_election.cc:288] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [CANDIDATE]: Term 1 election: Requested vote from peers 09e1ca7742db4ccbbce34d5d7e86e66a (127.13.148.2:37029), 572872fb23674259a8503fb35a977b3d (127.13.148.1:33681)
I20260926 22:48:58.961151 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68824e7ee00c4abc96fd1695c1147b7f" candidate_uuid: "339d79537ddf4f259e28b6b22e259932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a"
I20260926 22:48:58.961184 14619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68824e7ee00c4abc96fd1695c1147b7f" candidate_uuid: "339d79537ddf4f259e28b6b22e259932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572872fb23674259a8503fb35a977b3d"
I20260926 22:48:58.961228 14753 raft_consensus.cc:3037] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:48:58.961241 14619 raft_consensus.cc:3037] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:48:58.962078 14619 raft_consensus.cc:2454] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 339d79537ddf4f259e28b6b22e259932 in term 1.
I20260926 22:48:58.962078 14753 raft_consensus.cc:2454] T 68824e7ee00c4abc96fd1695c1147b7f P 09e1ca7742db4ccbbce34d5d7e86e66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 339d79537ddf4f259e28b6b22e259932 in term 1.
I20260926 22:48:58.962262 14820 leader_election.cc:302] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 339d79537ddf4f259e28b6b22e259932, 572872fb23674259a8503fb35a977b3d; no voters:
I20260926 22:48:58.962363 14952 raft_consensus.cc:2781] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:48:58.962503 14952 raft_consensus.cc:686] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [term 1 LEADER]: Becoming Leader. State: Replica: 339d79537ddf4f259e28b6b22e259932, State: Running, Role: LEADER
I20260926 22:48:58.962585 14952 consensus_queue.cc:237] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } }
I20260926 22:48:58.963323 14474 catalog_manager.cc:5495] T 68824e7ee00c4abc96fd1695c1147b7f P 339d79537ddf4f259e28b6b22e259932 reported cstate change: term changed from 0 to 1, leader changed from <none> to 339d79537ddf4f259e28b6b22e259932 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "339d79537ddf4f259e28b6b22e259932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "339d79537ddf4f259e28b6b22e259932" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 33667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09e1ca7742db4ccbbce34d5d7e86e66a" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 37029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572872fb23674259a8503fb35a977b3d" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 33681 } health_report { overall_health: UNKNOWN } } }
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:48:58.986127 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:58.979478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.13.148.1:33681)
0926 22:48:58.979532 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:58.979534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:58.979627 (+ 93us) server_negotiation.cc:407] Connection header received
0926 22:48:58.979695 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:58.979696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:58.979704 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:58.979743 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:58.980243 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.980706 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.981376 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:58.981555 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:58.981606 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:58.983660 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:58.983673 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:58.983673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:58.983680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:58.985085 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.985395 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.985398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.985399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.985449 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:58.985606 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:58.985607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:58.985607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:58.985763 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:58.985907 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:58.985966 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 22:48:58.986037 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:48:58 dist-test-slave-98hb krb5kdc[14429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:48:59.006613 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:59.000374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.13.148.1:33681)
0926 22:48:59.000431 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:59.000433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:59.000504 (+ 71us) server_negotiation.cc:407] Connection header received
0926 22:48:59.000569 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:59.000570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:59.000580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:59.000624 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:59.001129 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:59.001602 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:59.002336 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:59.002523 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:59.002576 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:59.004547 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:59.004548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:59.004548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:59.004556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:59.005578 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:59.005924 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:59.005926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:59.005926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:59.005975 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:59.006130 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:59.006131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:59.006131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:59.006216 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:59.006307 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:59.006457 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 22:48:59.006525 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 22:48:59 dist-test-slave-98hb krb5kdc[14429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260926 22:48:59.020845 14666 tablet.cc:2369] T 68824e7ee00c4abc96fd1695c1147b7f P 572872fb23674259a8503fb35a977b3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 22:48:59 dist-test-slave-98hb krb5kdc[14429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:48:59.026269 14946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:48:59.020087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.13.148.1:33681)
0926 22:48:59.020150 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:48:59.020152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:48:59.020232 (+ 80us) server_negotiation.cc:407] Connection header received
0926 22:48:59.020298 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:48:59.020299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:48:59.020308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:48:59.020354 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:48:59.020846 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:59.021336 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:59.022055 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:48:59.022282 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:48:59.022330 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:48:59.024244 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:48:59.024246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:48:59.024246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:48:59.024252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:48:59.025400 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:59.025670 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:59.025671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:59.025671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:59.025717 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:48:59.025918 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:48:59.025919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:48:59.025919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:48:59.026002 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:48:59.026064 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:48:59.026142 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 22:48:59.026195 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 22:48:59.026682 14579 tablet_service.cc:2908] Rejecting scan request for tablet 68824e7ee00c4abc96fd1695c1147b7f: Uninitialized: clean time has not yet been initialized
I20260926 22:48:59.027477 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 14536
I20260926 22:48:59.031379 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 14670
I20260926 22:48:59.035202 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 14804
I20260926 22:48:59.038987 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 14442
2026-09-26T22:48:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1885 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:48:59 dist-test-slave-98hb krb5kdc[14971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:48:59 dist-test-slave-98hb krb5kdc[14971](info): set up 2 sockets
Sep 26 22:48:59 dist-test-slave-98hb krb5kdc[14971](info): commencing operation
krb5kdc: starting...
W20260926 22:48:59.981370 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:00.088284 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46561
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:00.165859 14984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:00.166051 14984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:00.166071 14984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:00.167135 14984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:00.167168 14984 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 22:49:00.167179 14984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:00.167191 14984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:00.167201 14984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:00.167228 14984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:00.168601 14984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46561
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:00.168777 14984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:00.168944 14984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:00.173372 14990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:00.173409 14992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:00.173434 14989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:00.173556 14984 server_base.cc:1034] running on GCE node
I20260926 22:49:00.173856 14984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:00.174124 14984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:00.175254 14984 hybrid_clock.cc:648] HybridClock initialized: now 1790462940175240 us; error 32 us; skew 500 ppm
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:00.177528 14984 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:00.177942 14984 webserver.cc:460] Webserver started at http://127.13.148.62:42963/ using document root <none> and password file <none>
I20260926 22:49:00.178140 14984 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:00.178186 14984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:00.178311 14984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:00.179276 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "8b3538189e344eb0b7d12f728e62c5d9"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "9f7d52eed01600a989773ba250258687"
server_key_iv: "45d3a88d03e150ae61b89da35b2c15dc"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.179592 14984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "8b3538189e344eb0b7d12f728e62c5d9"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "9f7d52eed01600a989773ba250258687"
server_key_iv: "45d3a88d03e150ae61b89da35b2c15dc"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.180728 14984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:00.181303 14999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.181447 14984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:00.181524 14984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "8b3538189e344eb0b7d12f728e62c5d9"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "9f7d52eed01600a989773ba250258687"
server_key_iv: "45d3a88d03e150ae61b89da35b2c15dc"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.181582 14984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:00.187892 14984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:00.189831 14984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:00.189951 14984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:00.193920 14984 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46561
I20260926 22:49:00.193984 15061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46561 every 8 connection(s)
I20260926 22:49:00.194239 14984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:00.194828 15062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:00.196914 15062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9: Bootstrap starting.
I20260926 22:49:00.197438 15062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:00.197681 15062 log.cc:824] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:00.198299 15062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9: No bootstrap required, opened a new log
I20260926 22:49:00.200657 15062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3538189e344eb0b7d12f728e62c5d9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46561 } }
I20260926 22:49:00.200800 15062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:00.200821 15062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3538189e344eb0b7d12f728e62c5d9, State: Initialized, Role: FOLLOWER
I20260926 22:49:00.200958 15062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3538189e344eb0b7d12f728e62c5d9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46561 } }
I20260926 22:49:00.201017 15062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:00.201040 15062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:00.201083 15062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:00.201575 15062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3538189e344eb0b7d12f728e62c5d9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46561 } }
I20260926 22:49:00.201663 15062 leader_election.cc:302] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b3538189e344eb0b7d12f728e62c5d9; no voters:
I20260926 22:49:00.201848 15062 leader_election.cc:288] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:00.201884 15065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:00.202008 15065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 8b3538189e344eb0b7d12f728e62c5d9, State: Running, Role: LEADER
I20260926 22:49:00.202049 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 14984
I20260926 22:49:00.202078 15065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3538189e344eb0b7d12f728e62c5d9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46561 } }
I20260926 22:49:00.202128 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:00.202095 15062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:00.202291 13904 external_mini_cluster.cc:1428] Setting key b55778c4fa3c2a83a35d11887a0facad
I20260926 22:49:00.202870 15067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b3538189e344eb0b7d12f728e62c5d9. Latest consensus state: current_term: 1 leader_uuid: "8b3538189e344eb0b7d12f728e62c5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3538189e344eb0b7d12f728e62c5d9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46561 } } }
I20260926 22:49:00.202950 15067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:00.203132 15066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b3538189e344eb0b7d12f728e62c5d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b3538189e344eb0b7d12f728e62c5d9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46561 } } }
I20260926 22:49:00.203238 15066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:00.203374 15076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:00.204025 15076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:00.205487 15076 catalog_manager.cc:1347] Generated new cluster ID: 208b3b8ea90140468e16ccba5e1cadd8
I20260926 22:49:00.205535 15076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:00.213312 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.202892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.13.148.62:46561)
0926 22:49:00.203085 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.203095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.203129 (+ 34us) server_negotiation.cc:407] Connection header received
0926 22:49:00.203254 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.203261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.203319 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.204186 (+ 867us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:00.204285 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.205446 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.206523 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.207106 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.207200 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.209876 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:00.209880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:00.209880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:00.209892 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:00.211674 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.212083 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.212086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.212086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.212195 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.212442 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.212444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.212444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.212732 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:00.212887 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.212962 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.213095 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260926 22:49:00.223162 15076 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:00.223883 15076 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:00.228835 15076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9: Generated new TSK 0
I20260926 22:49:00.229009 15076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:00.256433 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46561
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:00.333961 15088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:00.334143 15088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:00.334173 15088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:00.335284 15088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:00.335325 15088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:00.335350 15088 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:00.335369 15088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:00.336757 15088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:00.336979 15088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:00.337176 15088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:00.341694 15094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:00.341706 15096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:00.341807 15088 server_base.cc:1034] running on GCE node
W20260926 22:49:00.341706 15093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:00.342093 15088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:00.342320 15088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:00.343456 15088 hybrid_clock.cc:648] HybridClock initialized: now 1790462940343443 us; error 27 us; skew 500 ppm
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:00.345723 15088 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:00.346125 15088 webserver.cc:460] Webserver started at http://127.13.148.1:40345/ using document root <none> and password file <none>
I20260926 22:49:00.346334 15088 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:00.346377 15088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:00.346493 15088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:00.347500 15088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "df854172ad2a44ca9db32d1b411a7fc5"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "1d68536fbb191ca4ec063c12627e449e"
server_key_iv: "ec385171c85bf4d408b89e9aec10bf54"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.347805 15088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "df854172ad2a44ca9db32d1b411a7fc5"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "1d68536fbb191ca4ec063c12627e449e"
server_key_iv: "ec385171c85bf4d408b89e9aec10bf54"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.348941 15088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:49:00.349517 15103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.349654 15088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.349762 15088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "df854172ad2a44ca9db32d1b411a7fc5"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "1d68536fbb191ca4ec063c12627e449e"
server_key_iv: "ec385171c85bf4d408b89e9aec10bf54"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.349828 15088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:00.385829 15088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:00.387516 15088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:00.387624 15088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:00.388078 15088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:00.388119 15088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.388140 15088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:00.388152 15088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.394466 15088 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:39127
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:00.394541 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:39127 every 8 connection(s)
I20260926 22:49:00.394857 15088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:00.397272 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.388506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:55119 (local address 127.13.148.62:46561)
0926 22:49:00.388564 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.388566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.388754 (+ 188us) server_negotiation.cc:407] Connection header received
0926 22:49:00.389226 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.389227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.389243 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.389315 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:00.390081 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.390834 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.392302 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.392603 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.392669 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.395001 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:00.395002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:00.395003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:00.395009 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:00.396280 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.396605 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.396606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.396606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.396650 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.396822 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.396822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.396822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.396911 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:00.396953 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.397150 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.397207 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:49:00.397450 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.388481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46561 (local address 127.13.148.1:55119)
0926 22:49:00.388626 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.388666 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.389128 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.389378 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.389382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.389711 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:00.389989 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.389996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.390928 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.390929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.392197 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.392201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.392686 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.392686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.392836 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.393362 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:00.393371 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:00.394897 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:00.396348 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.396352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.396354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.396549 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.396692 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.396692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.396692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.396778 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.396950 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:00.396953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:00.397057 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.397132 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.397251 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260926 22:49:00.397905 15219 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46561
I20260926 22:49:00.397981 15219 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:00.398142 15219 heartbeater.cc:511] Master 127.13.148.62:46561 requested a full tablet report, sending...
I20260926 22:49:00.398672 15015 ts_manager.cc:194] Registered new tserver with Master: df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127)
I20260926 22:49:00.399452 15015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:55119
I20260926 22:49:00.401153 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 15088
I20260926 22:49:00.401265 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:00.401436 13904 external_mini_cluster.cc:1428] Setting key 374279459133368ec62c163848546eb4
I20260926 22:49:00.403875 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.398824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46561 (local address 127.13.148.1:56857)
0926 22:49:00.398929 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.398938 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.398988 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.399087 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.399088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.399232 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:00.399277 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.399278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.399953 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.399954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.400541 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.400543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.400830 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.400831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.400881 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.401186 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:00.401190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:00.401521 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:00.402995 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.402996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.402996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.403195 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.403360 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.403361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.403361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.403424 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.403650 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:00.403650 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:00.403684 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.403738 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.403809 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":61,"threads_started":1}
I20260926 22:49:00.403920 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.398860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:56857 (local address 127.13.148.62:46561)
0926 22:49:00.398897 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.398899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.398991 (+ 92us) server_negotiation.cc:407] Connection header received
0926 22:49:00.399016 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.399016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.399025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.399084 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:00.399324 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.399875 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.400600 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.400772 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.400814 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.401661 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:00.401662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:00.401662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:00.401667 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:00.402932 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.403260 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.403260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.403261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.403309 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.403479 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.403480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.403481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.403597 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:00.403655 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.403771 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.403841 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:00.422602 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46561
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:00.499468 15228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:00.499665 15228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:00.499686 15228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:00.500842 15228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:00.500883 15228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:00.500903 15228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:00.500934 15228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:00.502465 15228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:00.502681 15228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:00.502897 15228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:00.507476 15234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:00.507576 15236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:00.507501 15228 server_base.cc:1034] running on GCE node
W20260926 22:49:00.507472 15233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:00.507912 15228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:00.508111 15228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:00.509241 15228 hybrid_clock.cc:648] HybridClock initialized: now 1790462940509230 us; error 27 us; skew 500 ppm
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:00.511463 15228 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:00.511873 15228 webserver.cc:460] Webserver started at http://127.13.148.2:46831/ using document root <none> and password file <none>
I20260926 22:49:00.512068 15228 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:00.512110 15228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:00.512223 15228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:00.513235 15228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "5327906a7552497f89d780049b9abbf6"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "69cfa56b7d1cd1ba38d8cde3b1fe3893"
server_key_iv: "bb77699323d4eda00d6e5f9866374578"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.513543 15228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "5327906a7552497f89d780049b9abbf6"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "69cfa56b7d1cd1ba38d8cde3b1fe3893"
server_key_iv: "bb77699323d4eda00d6e5f9866374578"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.514751 15228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:00.515340 15243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.515484 15228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:00.515559 15228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "5327906a7552497f89d780049b9abbf6"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "69cfa56b7d1cd1ba38d8cde3b1fe3893"
server_key_iv: "bb77699323d4eda00d6e5f9866374578"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.515614 15228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:00.532989 15228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:00.534704 15228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:00.534842 15228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:00.535318 15228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:00.535356 15228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.535385 15228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:00.535406 15228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:00.542361 15228 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:33209
I20260926 22:49:00.542438 15358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:33209 every 8 connection(s)
I20260926 22:49:00.542714 15228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:00.544374 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.535682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:36903 (local address 127.13.148.62:46561)
0926 22:49:00.535751 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.535753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.535945 (+ 192us) server_negotiation.cc:407] Connection header received
0926 22:49:00.536585 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.536586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.536598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.536657 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:00.537477 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.538094 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.539522 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.539770 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.539823 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.542046 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:00.542048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:00.542048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:00.542055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:00.543401 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.543721 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.543722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.543722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.543768 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.543930 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.543930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.543930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.544018 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:00.544061 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.544259 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.544313 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:49:00.544518 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.535665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46561 (local address 127.13.148.2:36903)
0926 22:49:00.535812 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.535859 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.536492 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.536748 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.536752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.537094 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:00.537384 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.537392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.538161 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.538162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.539420 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.539423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.539841 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.539843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.539983 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.540428 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:00.540439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:00.541955 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:00.543469 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.543472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.543473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.543664 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.543813 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.543813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.543814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.543892 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.544062 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:00.544064 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:00.544163 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.544236 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.544346 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260926 22:49:00.544973 15359 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46561
I20260926 22:49:00.545047 15359 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:00.545212 15359 heartbeater.cc:511] Master 127.13.148.62:46561 requested a full tablet report, sending...
I20260926 22:49:00.545576 15016 ts_manager.cc:194] Registered new tserver with Master: 5327906a7552497f89d780049b9abbf6 (127.13.148.2:33209)
I20260926 22:49:00.546031 15016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:36903
I20260926 22:49:00.546496 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 15228
I20260926 22:49:00.546573 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:00.546720 13904 external_mini_cluster.cc:1428] Setting key 43e58f415736fb9012f2e7c99bd412b9
I20260926 22:49:00.551092 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.545929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:35985 (local address 127.13.148.62:46561)
0926 22:49:00.546141 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.546143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.546150 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:00.546166 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.546167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.546176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.546244 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:00.546604 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.547261 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.547903 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.548092 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.548143 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.548969 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:00.548970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:00.548970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:00.548975 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:00.550114 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.550459 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.550460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.550460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.550504 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.550671 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.550671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.550672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.550790 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:00.550837 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.550987 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.551038 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260926 22:49:00.551180 15365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.545880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46561 (local address 127.13.148.2:35985)
0926 22:49:00.545987 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.545996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.546037 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.546254 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.546255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.546439 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:00.546516 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.546518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.547349 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.547349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.547835 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.547836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.548148 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.548149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.548218 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.548555 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:00.548559 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:00.548892 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:00.550195 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.550196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.550197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.550395 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.550556 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.550556 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.550557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.550619 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.550848 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:00.550849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:00.550895 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.550973 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.551066 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:00.568305 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46561
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:00.647368 15368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:00.647562 15368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:00.647593 15368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:00.648731 15368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:00.648772 15368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:00.648798 15368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:00.648821 15368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:00.650240 15368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:46561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:00.650461 15368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:00.650661 15368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:00.655314 15376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:00.655375 15373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:00.655434 15368 server_base.cc:1034] running on GCE node
W20260926 22:49:00.655508 15374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:00.655885 15368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:00.656155 15368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:00.657291 15368 hybrid_clock.cc:648] HybridClock initialized: now 1790462940657280 us; error 28 us; skew 500 ppm
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:00.659579 15368 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:00.659972 15368 webserver.cc:460] Webserver started at http://127.13.148.3:44539/ using document root <none> and password file <none>
I20260926 22:49:00.660151 15368 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:00.660192 15368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:00.660296 15368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:00.661267 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "6f00415f95be4c59be4a9f13cf8293e1"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "b7608bdbde2ed0f93c5bb7053ba0a3f3"
server_key_iv: "fdbf0c1d684eaa737b98bb00093de802"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.661571 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "6f00415f95be4c59be4a9f13cf8293e1"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "b7608bdbde2ed0f93c5bb7053ba0a3f3"
server_key_iv: "fdbf0c1d684eaa737b98bb00093de802"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.662751 15368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 22:49:00.663370 15383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.663496 15368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:00.663573 15368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "6f00415f95be4c59be4a9f13cf8293e1"
format_stamp: "Formatted at 2026-09-26 22:49:00 on dist-test-slave-98hb"
server_key: "b7608bdbde2ed0f93c5bb7053ba0a3f3"
server_key_iv: "fdbf0c1d684eaa737b98bb00093de802"
server_key_version: "encryptionkey@0"
I20260926 22:49:00.663630 15368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:00.684721 15368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:00.686460 15368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:00.686589 15368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:00.687068 15368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:00.687106 15368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.687135 15368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:00.687152 15368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:00.693838 15368 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:39059
I20260926 22:49:00.694151 15368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:00.694185 15498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:39059 every 8 connection(s)
I20260926 22:49:00.695959 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.687479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:39929 (local address 127.13.148.62:46561)
0926 22:49:00.687541 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.687543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.687718 (+ 175us) server_negotiation.cc:407] Connection header received
0926 22:49:00.688328 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.688330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.688343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.688453 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:00.689262 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.690032 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.691083 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.691290 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.691343 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.693446 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:00.693447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:00.693448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:00.693454 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:00.694786 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.695127 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.695128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.695128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.695190 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.695393 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.695394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.695394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.695502 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:00.695556 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.695802 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.695874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:49:00.696121 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.687445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46561 (local address 127.13.148.3:39929)
0926 22:49:00.687588 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.687626 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.688212 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.688546 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.688551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.688888 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:00.689158 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.689164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.690125 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.690126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.691006 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.691009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.691366 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.691367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.691471 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.691922 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:00.691933 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:00.693364 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:00.694852 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.694858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.694860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.695064 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.695242 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.695242 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.695243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.695341 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.695552 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:00.695558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:00.695690 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.695773 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.695914 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260926 22:49:00.696501 15499 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46561
I20260926 22:49:00.696571 15499 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:00.696733 15499 heartbeater.cc:511] Master 127.13.148.62:46561 requested a full tablet report, sending...
I20260926 22:49:00.697067 15015 ts_manager.cc:194] Registered new tserver with Master: 6f00415f95be4c59be4a9f13cf8293e1 (127.13.148.3:39059)
I20260926 22:49:00.697428 15015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:39929
I20260926 22:49:00.701922 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.697428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58035 (local address 127.13.148.62:46561)
0926 22:49:00.697467 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.697468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.697533 (+ 65us) server_negotiation.cc:407] Connection header received
0926 22:49:00.697587 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.697587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.697594 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.697636 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:00.697977 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.698494 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.699128 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.699308 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.699352 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.699991 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:00.699992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:00.699992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:00.699997 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:00.701133 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.701373 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.701373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.701374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.701413 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.701560 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.701561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.701561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.701647 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:00.701695 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.701818 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.701868 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:00.701936 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.697366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46561 (local address 127.13.148.3:58035)
0926 22:49:00.697483 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.697493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.697545 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.697640 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.697641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.697832 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:00.697887 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.697889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.698568 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.698569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.699070 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.699072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.699356 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.699356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.699405 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.699700 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:00.699704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:00.699932 (+ 228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:00.701195 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.701195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.701195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.701330 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.701452 (+ 122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:00.701453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:00.701453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:00.701511 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:00.701691 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:00.701692 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:00.701724 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.701802 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.701871 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260926 22:49:00.702320 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 15368
I20260926 22:49:00.702406 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:00.702579 13904 external_mini_cluster.cc:1428] Setting key 9d4aa1f1f404fad316719d2f118a89d9
I20260926 22:49:00.703373 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:49:00 dist-test-slave-98hb krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:00.723382 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.717269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.13.148.62:46561)
0926 22:49:00.717329 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.717331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.717338 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:00.717356 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.717357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.717364 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.717405 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:00.717980 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.718458 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.719251 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.719436 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.719488 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.721418 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:00.721419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:00.721420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:00.721426 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:00.722466 (+ 1040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.722744 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.722746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.722746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.722791 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:00.722944 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:00.722945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:00.722945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:00.723039 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:00.723119 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.723214 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.723284 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:49:00.724709 15015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:00.726234 15015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:00.731938 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.728062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:39127 (local address 127.0.0.1:47464)
0926 22:49:00.728214 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.728243 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.728308 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.728693 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.728696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.728700 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:00.728764 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.728767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.730100 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.730101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.731011 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.731013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.731731 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.731731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.731793 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.731804 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.731851 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.731887 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":28,"thread_start_us":56,"threads_started":1}
I20260926 22:49:00.731994 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.728377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.13.148.1:39127)
0926 22:49:00.728550 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.728553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.728561 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:00.728592 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.728594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.728607 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.728678 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:00.728861 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.730018 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.731075 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.731655 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.731718 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.731772 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.731863 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.731900 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":68,"threads_started":1}
I20260926 22:49:00.732358 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.728274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:39059 (local address 127.0.0.1:38940)
0926 22:49:00.728750 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.728759 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.728807 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.729001 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.729002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.729003 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:00.729042 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.729043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.729991 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.729993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.731304 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.731306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.732124 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.732125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.732182 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.732186 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.732231 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.732269 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":8,"thread_start_us":32,"threads_started":1}
I20260926 22:49:00.732604 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.728373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.13.148.3:39059)
0926 22:49:00.728515 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.728519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.728810 (+ 291us) server_negotiation.cc:407] Connection header received
0926 22:49:00.728857 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.728859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.728873 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.728941 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:00.729115 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.729915 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.731382 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.732060 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.732121 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.732174 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.732482 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.732521 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260926 22:49:00.732694 15153 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2081df03ac4733bd48ef06f8afb8d0 (DEFAULT_TABLE table=test-table [id=572ae68cfa5a447cab4ebe570979c015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:00.733079 15153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b2081df03ac4733bd48ef06f8afb8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:00.733265 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.728192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.13.148.2:33209)
0926 22:49:00.728527 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.728532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.728541 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:00.728581 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.728583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.728598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.728675 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:00.728907 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.730026 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.732209 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.732908 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.732995 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.733048 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.733143 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.733177 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":85,"threads_started":1}
I20260926 22:49:00.733173 15433 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2081df03ac4733bd48ef06f8afb8d0 (DEFAULT_TABLE table=test-table [id=572ae68cfa5a447cab4ebe570979c015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:00.733492 15433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b2081df03ac4733bd48ef06f8afb8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:00.733604 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.728081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33209 (local address 127.0.0.1:38014)
0926 22:49:00.728359 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.728369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.728417 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.728693 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.728696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.728698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:00.728764 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.728766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.730181 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.730182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.731303 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.731306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.733015 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.733016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.733075 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.733079 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.733505 (+ 426us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.733541 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":15,"thread_start_us":53,"threads_started":1}
I20260926 22:49:00.736452 15519 tablet_bootstrap.cc:492] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: Bootstrap starting.
I20260926 22:49:00.736837 15520 tablet_bootstrap.cc:492] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Bootstrap starting.
I20260926 22:49:00.737136 15284 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2081df03ac4733bd48ef06f8afb8d0 (DEFAULT_TABLE table=test-table [id=572ae68cfa5a447cab4ebe570979c015]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:00.737327 15519 tablet_bootstrap.cc:654] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:00.737329 15520 tablet_bootstrap.cc:654] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:00.737449 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b2081df03ac4733bd48ef06f8afb8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:00.737629 15520 log.cc:824] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:00.737643 15519 log.cc:824] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:00.738333 15520 tablet_bootstrap.cc:492] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: No bootstrap required, opened a new log
I20260926 22:49:00.738392 15520 ts_tablet_manager.cc:1397] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:00.738549 15519 tablet_bootstrap.cc:492] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: No bootstrap required, opened a new log
I20260926 22:49:00.738617 15519 ts_tablet_manager.cc:1397] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:00.739879 15520 raft_consensus.cc:348] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.740033 15520 raft_consensus.cc:374] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:00.740067 15520 raft_consensus.cc:729] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df854172ad2a44ca9db32d1b411a7fc5, State: Initialized, Role: FOLLOWER
I20260926 22:49:00.740218 15520 consensus_queue.cc:260] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.740473 15520 ts_tablet_manager.cc:1428] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:00.740584 15219 heartbeater.cc:503] Master 127.13.148.62:46561 was elected leader, sending a full tablet report...
I20260926 22:49:00.740482 15519 raft_consensus.cc:348] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.740626 15519 raft_consensus.cc:374] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:00.740666 15519 raft_consensus.cc:729] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f00415f95be4c59be4a9f13cf8293e1, State: Initialized, Role: FOLLOWER
I20260926 22:49:00.740801 15519 consensus_queue.cc:260] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.741046 15519 ts_tablet_manager.cc:1428] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:49:00.741072 15499 heartbeater.cc:503] Master 127.13.148.62:46561 was elected leader, sending a full tablet report...
I20260926 22:49:00.741282 15524 tablet_bootstrap.cc:492] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Bootstrap starting.
I20260926 22:49:00.741930 15524 tablet_bootstrap.cc:654] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:00.742198 15524 log.cc:824] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:00.742769 15524 tablet_bootstrap.cc:492] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: No bootstrap required, opened a new log
I20260926 22:49:00.742820 15524 ts_tablet_manager.cc:1397] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:00.744242 15524 raft_consensus.cc:348] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.744361 15524 raft_consensus.cc:374] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:00.744381 15524 raft_consensus.cc:729] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5327906a7552497f89d780049b9abbf6, State: Initialized, Role: FOLLOWER
I20260926 22:49:00.744483 15524 consensus_queue.cc:260] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.744688 15524 ts_tablet_manager.cc:1428] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:00.744740 15359 heartbeater.cc:503] Master 127.13.148.62:46561 was elected leader, sending a full tablet report...
W20260926 22:49:00.793274 15360 tablet.cc:2369] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:49:00.895555 15220 tablet.cc:2369] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:00.898953 15527 raft_consensus.cc:482] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:00.899058 15527 raft_consensus.cc:504] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.899354 15527 leader_election.cc:288] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127), 6f00415f95be4c59be4a9f13cf8293e1 (127.13.148.3:39059)
I20260926 22:49:00.902070 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.899544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:39127 (local address 127.13.148.2:60947)
0926 22:49:00.899601 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.899613 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.899675 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.899868 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.899869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.899871 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:00.899926 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.899928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.900561 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.900562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.901307 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.901309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.901818 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.901819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.901897 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.901900 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.901945 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.901999 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 22:49:00.902070 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.899592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:60947 (local address 127.13.148.1:39127)
0926 22:49:00.899643 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.899645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.899658 (+ 13us) server_negotiation.cc:407] Connection header received
0926 22:49:00.899736 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.899737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.899745 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.899805 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:00.900002 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.900489 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.901389 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.901758 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.901820 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.901829 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.901954 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.901986 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:49:00.902175 15517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.899772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:50455 (local address 127.13.148.3:39059)
0926 22:49:00.899835 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.899837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.899866 (+ 29us) server_negotiation.cc:407] Connection header received
0926 22:49:00.899917 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.899918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.899928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.899972 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:00.900101 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.900559 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.901449 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.901854 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.901913 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.901920 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.902080 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.902114 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 22:49:00.902194 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.899671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:39059 (local address 127.13.148.2:50455)
0926 22:49:00.899816 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:00.899824 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:00.899871 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:00.899974 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:00.899975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:00.899976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:00.900044 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.900046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.900631 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.900631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.901356 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:00.901358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.901920 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.901921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:00.902026 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.902030 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:00.902076 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:49:00.902111 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":12,"thread_start_us":96,"threads_started":1}
I20260926 22:49:00.902377 15173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2081df03ac4733bd48ef06f8afb8d0" candidate_uuid: "5327906a7552497f89d780049b9abbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df854172ad2a44ca9db32d1b411a7fc5" is_pre_election: true
I20260926 22:49:00.902539 15173 raft_consensus.cc:2454] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5327906a7552497f89d780049b9abbf6 in term 0.
I20260926 22:49:00.902617 15453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2081df03ac4733bd48ef06f8afb8d0" candidate_uuid: "5327906a7552497f89d780049b9abbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f00415f95be4c59be4a9f13cf8293e1" is_pre_election: true
I20260926 22:49:00.902735 15245 leader_election.cc:302] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5327906a7552497f89d780049b9abbf6, df854172ad2a44ca9db32d1b411a7fc5; no voters:
I20260926 22:49:00.902757 15453 raft_consensus.cc:2454] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5327906a7552497f89d780049b9abbf6 in term 0.
I20260926 22:49:00.902873 15527 raft_consensus.cc:2781] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:00.902926 15527 raft_consensus.cc:482] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:00.902954 15527 raft_consensus.cc:3037] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:00.903904 15527 raft_consensus.cc:504] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.904031 15527 leader_election.cc:288] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [CANDIDATE]: Term 1 election: Requested vote from peers df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127), 6f00415f95be4c59be4a9f13cf8293e1 (127.13.148.3:39059)
I20260926 22:49:00.904170 15453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2081df03ac4733bd48ef06f8afb8d0" candidate_uuid: "5327906a7552497f89d780049b9abbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f00415f95be4c59be4a9f13cf8293e1"
I20260926 22:49:00.904229 15453 raft_consensus.cc:3037] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:00.904892 15453 raft_consensus.cc:2454] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5327906a7552497f89d780049b9abbf6 in term 1.
I20260926 22:49:00.904986 15173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2081df03ac4733bd48ef06f8afb8d0" candidate_uuid: "5327906a7552497f89d780049b9abbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df854172ad2a44ca9db32d1b411a7fc5"
I20260926 22:49:00.905027 15245 leader_election.cc:302] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5327906a7552497f89d780049b9abbf6, 6f00415f95be4c59be4a9f13cf8293e1; no voters:
I20260926 22:49:00.905056 15173 raft_consensus.cc:3037] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:00.905113 15527 raft_consensus.cc:2781] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:00.905251 15527 raft_consensus.cc:686] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 1 LEADER]: Becoming Leader. State: Replica: 5327906a7552497f89d780049b9abbf6, State: Running, Role: LEADER
I20260926 22:49:00.905344 15527 consensus_queue.cc:237] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } }
I20260926 22:49:00.905779 15173 raft_consensus.cc:2454] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5327906a7552497f89d780049b9abbf6 in term 1.
I20260926 22:49:00.906122 15016 catalog_manager.cc:5495] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5327906a7552497f89d780049b9abbf6 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "5327906a7552497f89d780049b9abbf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } health_report { overall_health: HEALTHY } } }
I20260926 22:49:00.911157 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:00.909076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.13.148.2:33209)
0926 22:49:00.909112 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:00.909114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:00.909160 (+ 46us) server_negotiation.cc:407] Connection header received
0926 22:49:00.909225 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:00.909225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:00.909238 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:00.909289 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:00.909390 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.909862 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.910584 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:00.910749 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:00.910788 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:00.910944 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:00.911024 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:00.911072 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:00.911099 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 22:49:00.911112 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:49:00.912691 15453 raft_consensus.cc:1261] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Refusing update from remote peer 5327906a7552497f89d780049b9abbf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:00.912689 15173 raft_consensus.cc:1261] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Refusing update from remote peer 5327906a7552497f89d780049b9abbf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:00.912945 15529 consensus_queue.cc:1035] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:00.913054 15527 consensus_queue.cc:1035] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:00.915475 15534 mvcc.cc:205] Tried to move back new op lower bound from 7333736205975990272 to 7333736205948731392. Current Snapshot: MvccSnapshot[applied={T|T < 7333736205975990272}]
I20260926 22:49:00.916561 15533 mvcc.cc:205] Tried to move back new op lower bound from 7333736205975990272 to 7333736205948731392. Current Snapshot: MvccSnapshot[applied={T|T < 7333736205975990272 or (T in {7333736205975990272})}]
I20260926 22:49:00.918097 15016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:00.918200 15016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:00.918906 15016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9: Sending DeleteTablet for 3 replicas of tablet 8b2081df03ac4733bd48ef06f8afb8d0
I20260926 22:49:00.919215 15433 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b2081df03ac4733bd48ef06f8afb8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:00 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:38940
I20260926 22:49:00.919421 15153 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b2081df03ac4733bd48ef06f8afb8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:00 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:47464
I20260926 22:49:00.919441 15016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:00.919565 15016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:00.919625 15284 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b2081df03ac4733bd48ef06f8afb8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:00 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:38014
I20260926 22:49:00.919813 15544 tablet_replica.cc:330] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: stopping tablet replica
I20260926 22:49:00.919821 15542 tablet_replica.cc:330] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: stopping tablet replica
I20260926 22:49:00.919910 15542 raft_consensus.cc:2229] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:00.919984 15544 raft_consensus.cc:2229] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:00.920018 15542 raft_consensus.cc:2258] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:00.920135 15544 raft_consensus.cc:2258] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:00.920451 15542 ts_tablet_manager.cc:1905] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:00.920635 15544 ts_tablet_manager.cc:1905] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:00.920846 15153 tablet_service.cc:1467] Processing CreateTablet for tablet 7077f7bc971642bb96f41a38d40e1908 (DEFAULT_TABLE table=test-table [id=afd7f8e40f00484b955ef2e2ef24e515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:00.920969 15153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077f7bc971642bb96f41a38d40e1908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:00.922226 15520 tablet_bootstrap.cc:492] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: Bootstrap starting.
I20260926 22:49:00.922407 15544 ts_tablet_manager.cc:1918] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:00.922503 15544 log.cc:1196] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/8b2081df03ac4733bd48ef06f8afb8d0
I20260926 22:49:00.922621 15520 tablet_bootstrap.cc:654] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:00.922801 15544 ts_tablet_manager.cc:1939] T 8b2081df03ac4733bd48ef06f8afb8d0 P 5327906a7552497f89d780049b9abbf6: Deleting consensus metadata
I20260926 22:49:00.923070 15433 tablet_service.cc:1467] Processing CreateTablet for tablet 7077f7bc971642bb96f41a38d40e1908 (DEFAULT_TABLE table=test-table [id=afd7f8e40f00484b955ef2e2ef24e515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:00.923261 15000 catalog_manager.cc:4841] TS 5327906a7552497f89d780049b9abbf6 (127.13.148.2:33209): tablet 8b2081df03ac4733bd48ef06f8afb8d0 (table test-table [id=572ae68cfa5a447cab4ebe570979c015]) successfully deleted
I20260926 22:49:00.923359 15520 tablet_bootstrap.cc:492] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: No bootstrap required, opened a new log
I20260926 22:49:00.923404 15520 ts_tablet_manager.cc:1397] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:00.923444 15284 tablet_service.cc:1467] Processing CreateTablet for tablet 7077f7bc971642bb96f41a38d40e1908 (DEFAULT_TABLE table=test-table [id=afd7f8e40f00484b955ef2e2ef24e515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:00.923532 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077f7bc971642bb96f41a38d40e1908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:00.923563 15520 raft_consensus.cc:348] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:00.923671 15520 raft_consensus.cc:374] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:00.923731 15520 raft_consensus.cc:729] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df854172ad2a44ca9db32d1b411a7fc5, State: Initialized, Role: FOLLOWER
I20260926 22:49:00.923782 15520 consensus_queue.cc:260] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:00.923905 15520 ts_tablet_manager.cc:1428] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.924569 15524 tablet_bootstrap.cc:492] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: Bootstrap starting.
I20260926 22:49:00.924981 15524 tablet_bootstrap.cc:654] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:00.925443 15433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7077f7bc971642bb96f41a38d40e1908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:00.925508 15543 tablet_replica.cc:330] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: stopping tablet replica
I20260926 22:49:00.925628 15543 raft_consensus.cc:2229] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:00.925755 15543 raft_consensus.cc:2258] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:00.925443 15542 ts_tablet_manager.cc:1918] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:00.925912 15542 log.cc:1196] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/8b2081df03ac4733bd48ef06f8afb8d0
I20260926 22:49:00.926152 15542 ts_tablet_manager.cc:1939] T 8b2081df03ac4733bd48ef06f8afb8d0 P 6f00415f95be4c59be4a9f13cf8293e1: Deleting consensus metadata
I20260926 22:49:00.926558 15001 catalog_manager.cc:4841] TS 6f00415f95be4c59be4a9f13cf8293e1 (127.13.148.3:39059): tablet 8b2081df03ac4733bd48ef06f8afb8d0 (table test-table [id=572ae68cfa5a447cab4ebe570979c015]) successfully deleted
I20260926 22:49:00.926645 15519 tablet_bootstrap.cc:492] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: Bootstrap starting.
I20260926 22:49:00.926761 15524 tablet_bootstrap.cc:492] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: No bootstrap required, opened a new log
I20260926 22:49:00.926800 15524 ts_tablet_manager.cc:1397] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:00.926947 15524 raft_consensus.cc:348] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:00.927004 15519 tablet_bootstrap.cc:654] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:00.927021 15524 raft_consensus.cc:374] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:00.927042 15524 raft_consensus.cc:729] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5327906a7552497f89d780049b9abbf6, State: Initialized, Role: FOLLOWER
I20260926 22:49:00.927080 15524 consensus_queue.cc:260] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:00.927168 15524 ts_tablet_manager.cc:1428] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:00.927310 15543 ts_tablet_manager.cc:1905] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:00.927691 15519 tablet_bootstrap.cc:492] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: No bootstrap required, opened a new log
I20260926 22:49:00.927740 15519 ts_tablet_manager.cc:1397] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:00.927871 15519 raft_consensus.cc:348] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:00.927940 15519 raft_consensus.cc:374] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:00.927958 15519 raft_consensus.cc:729] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f00415f95be4c59be4a9f13cf8293e1, State: Initialized, Role: FOLLOWER
I20260926 22:49:00.928002 15519 consensus_queue.cc:260] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:00.928112 15519 ts_tablet_manager.cc:1428] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:00.929023 15543 ts_tablet_manager.cc:1918] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:00.929077 15543 log.cc:1196] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/8b2081df03ac4733bd48ef06f8afb8d0
I20260926 22:49:00.929229 15543 ts_tablet_manager.cc:1939] T 8b2081df03ac4733bd48ef06f8afb8d0 P df854172ad2a44ca9db32d1b411a7fc5: Deleting consensus metadata
I20260926 22:49:00.929602 15001 catalog_manager.cc:4841] TS df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127): tablet 8b2081df03ac4733bd48ef06f8afb8d0 (table test-table [id=572ae68cfa5a447cab4ebe570979c015]) successfully deleted
W20260926 22:49:00.944681 15500 tablet.cc:2369] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:01.123796 15529 raft_consensus.cc:482] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:01.123898 15529 raft_consensus.cc:504] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:01.124032 15529 leader_election.cc:288] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127), 6f00415f95be4c59be4a9f13cf8293e1 (127.13.148.3:39059)
I20260926 22:49:01.124272 15173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077f7bc971642bb96f41a38d40e1908" candidate_uuid: "5327906a7552497f89d780049b9abbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df854172ad2a44ca9db32d1b411a7fc5" is_pre_election: true
I20260926 22:49:01.124318 15453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077f7bc971642bb96f41a38d40e1908" candidate_uuid: "5327906a7552497f89d780049b9abbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f00415f95be4c59be4a9f13cf8293e1" is_pre_election: true
I20260926 22:49:01.124370 15173 raft_consensus.cc:2454] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5327906a7552497f89d780049b9abbf6 in term 0.
I20260926 22:49:01.124402 15453 raft_consensus.cc:2454] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5327906a7552497f89d780049b9abbf6 in term 0.
I20260926 22:49:01.124524 15245 leader_election.cc:302] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5327906a7552497f89d780049b9abbf6, df854172ad2a44ca9db32d1b411a7fc5; no voters:
I20260926 22:49:01.124629 15529 raft_consensus.cc:2781] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:01.124677 15529 raft_consensus.cc:482] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:01.124703 15529 raft_consensus.cc:3037] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:01.125401 15529 raft_consensus.cc:504] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:01.125501 15529 leader_election.cc:288] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [CANDIDATE]: Term 1 election: Requested vote from peers df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127), 6f00415f95be4c59be4a9f13cf8293e1 (127.13.148.3:39059)
I20260926 22:49:01.125659 15173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077f7bc971642bb96f41a38d40e1908" candidate_uuid: "5327906a7552497f89d780049b9abbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df854172ad2a44ca9db32d1b411a7fc5"
I20260926 22:49:01.125692 15453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7077f7bc971642bb96f41a38d40e1908" candidate_uuid: "5327906a7552497f89d780049b9abbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f00415f95be4c59be4a9f13cf8293e1"
I20260926 22:49:01.125758 15173 raft_consensus.cc:3037] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:01.125797 15453 raft_consensus.cc:3037] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:01.126360 15453 raft_consensus.cc:2454] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5327906a7552497f89d780049b9abbf6 in term 1.
I20260926 22:49:01.126360 15173 raft_consensus.cc:2454] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5327906a7552497f89d780049b9abbf6 in term 1.
I20260926 22:49:01.126515 15245 leader_election.cc:302] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5327906a7552497f89d780049b9abbf6, 6f00415f95be4c59be4a9f13cf8293e1; no voters:
I20260926 22:49:01.126602 15529 raft_consensus.cc:2781] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:01.126653 15529 raft_consensus.cc:686] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 1 LEADER]: Becoming Leader. State: Replica: 5327906a7552497f89d780049b9abbf6, State: Running, Role: LEADER
I20260926 22:49:01.126691 15529 consensus_queue.cc:237] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } }
I20260926 22:49:01.127197 15016 catalog_manager.cc:5495] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5327906a7552497f89d780049b9abbf6 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "5327906a7552497f89d780049b9abbf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } health_report { overall_health: UNKNOWN } } }
Sep 26 22:49:01 dist-test-slave-98hb krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:01.180539 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.174383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46561 (local address 127.0.0.1:51984)
0926 22:49:01.174537 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:01.174548 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:01.174615 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:01.174736 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:01.174736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:01.175038 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:01.175086 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.175089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.175704 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.175705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.176234 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.176236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.176532 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.176533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.176578 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.177025 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:01.177029 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:01.178421 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:01.179697 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:01.179704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:01.179706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:01.179903 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:01.180043 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:01.180044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:01.180044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:01.180094 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:01.180263 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:01.180267 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:01.180306 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 22:49:01.180351 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:49:01.180442 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260926 22:49:01.180577 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.174438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.13.148.62:46561)
0926 22:49:01.174494 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:01.174497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:01.174608 (+ 111us) server_negotiation.cc:407] Connection header received
0926 22:49:01.174665 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:01.174665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:01.174675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:01.174726 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:01.175164 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.175637 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.176308 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.176478 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.176530 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.178514 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:01.178515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:01.178515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:01.178522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:01.179625 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:01.179957 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:01.179958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:01.179958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:01.180002 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:01.180132 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:01.180132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:01.180132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:01.180224 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:01.180267 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:01.180371 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 22:49:01.180481 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":29}
I20260926 22:49:01.182269 15016 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51984:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 22:49:01.182462 15016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:01.183681 15284 tablet_service.cc:1467] Processing CreateTablet for tablet 022c1de9cdf94a25b98265888b0c201d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fb3aa531ab143b09714412aa0ca6a45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:01.183679 15433 tablet_service.cc:1467] Processing CreateTablet for tablet 022c1de9cdf94a25b98265888b0c201d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fb3aa531ab143b09714412aa0ca6a45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:01.183743 15153 tablet_service.cc:1467] Processing CreateTablet for tablet 022c1de9cdf94a25b98265888b0c201d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fb3aa531ab143b09714412aa0ca6a45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:01.183828 15284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022c1de9cdf94a25b98265888b0c201d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:01.183853 15433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022c1de9cdf94a25b98265888b0c201d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:01.183862 15153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022c1de9cdf94a25b98265888b0c201d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:01.184938 15524 tablet_bootstrap.cc:492] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: Bootstrap starting.
I20260926 22:49:01.185252 15520 tablet_bootstrap.cc:492] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: Bootstrap starting.
I20260926 22:49:01.185276 15524 tablet_bootstrap.cc:654] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:01.185276 15519 tablet_bootstrap.cc:492] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: Bootstrap starting.
I20260926 22:49:01.185645 15520 tablet_bootstrap.cc:654] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:01.185689 15519 tablet_bootstrap.cc:654] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:01.186215 15520 tablet_bootstrap.cc:492] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: No bootstrap required, opened a new log
I20260926 22:49:01.186261 15520 ts_tablet_manager.cc:1397] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:01.186319 15519 tablet_bootstrap.cc:492] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: No bootstrap required, opened a new log
I20260926 22:49:01.186357 15519 ts_tablet_manager.cc:1397] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:01.186384 15524 tablet_bootstrap.cc:492] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: No bootstrap required, opened a new log
I20260926 22:49:01.186419 15524 ts_tablet_manager.cc:1397] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:01.186409 15520 raft_consensus.cc:348] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.186472 15520 raft_consensus.cc:374] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:01.186491 15520 raft_consensus.cc:729] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df854172ad2a44ca9db32d1b411a7fc5, State: Initialized, Role: FOLLOWER
I20260926 22:49:01.186494 15519 raft_consensus.cc:348] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.186506 15524 raft_consensus.cc:348] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.186546 15519 raft_consensus.cc:374] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:01.186558 15524 raft_consensus.cc:374] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:01.186537 15520 consensus_queue.cc:260] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.186565 15519 raft_consensus.cc:729] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f00415f95be4c59be4a9f13cf8293e1, State: Initialized, Role: FOLLOWER
I20260926 22:49:01.186578 15524 raft_consensus.cc:729] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5327906a7552497f89d780049b9abbf6, State: Initialized, Role: FOLLOWER
I20260926 22:49:01.186604 15519 consensus_queue.cc:260] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.186626 15520 ts_tablet_manager.cc:1428] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:01.186617 15524 consensus_queue.cc:260] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.186679 15519 ts_tablet_manager.cc:1428] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:01.186695 15524 ts_tablet_manager.cc:1428] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:01.186693 15523 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.186786 15523 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:01.186843 15525 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.186913 15525 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:01.186975 15529 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.187044 15529 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:01.434525 15525 raft_consensus.cc:482] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:01.434619 15525 raft_consensus.cc:504] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.434954 15525 leader_election.cc:288] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5327906a7552497f89d780049b9abbf6 (127.13.148.2:33209), df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127)
I20260926 22:49:01.437752 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.435182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:43765 (local address 127.13.148.1:39127)
0926 22:49:01.435290 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:01.435292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:01.435361 (+ 69us) server_negotiation.cc:407] Connection header received
0926 22:49:01.435415 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:01.435416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:01.435426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:01.435482 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:01.435620 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.436122 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.437048 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.437427 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.437488 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.437497 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:01.437615 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 22:49:01.437650 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":46,"threads_started":1}
I20260926 22:49:01.437752 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.435191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:38549 (local address 127.13.148.2:33209)
0926 22:49:01.435300 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:01.435302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:01.435333 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:49:01.435413 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:01.435414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:01.435424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:01.435476 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:01.435636 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.436121 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.437053 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.437419 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.437478 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.437488 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:01.437622 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 22:49:01.437661 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":42,"threads_started":1}
I20260926 22:49:01.437754 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.435127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33209 (local address 127.13.148.3:38549)
0926 22:49:01.435270 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:01.435281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:01.435343 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:01.435486 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:01.435486 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:01.435489 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:01.435575 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.435576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.436192 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.436193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.436967 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.436969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.437488 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.437489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.437555 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.437559 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:01.437608 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:49:01.437651 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260926 22:49:01.437762 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.435127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:39127 (local address 127.13.148.3:43765)
0926 22:49:01.435276 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:01.435286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:01.435364 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:01.435492 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:01.435492 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:01.435494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:01.435564 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.435566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.436196 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.436197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.436969 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.436971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.437496 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.437497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.437555 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.437558 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:01.437603 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:49:01.437648 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":83,"thread_start_us":53,"threads_started":1}
I20260926 22:49:01.437984 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022c1de9cdf94a25b98265888b0c201d" candidate_uuid: "6f00415f95be4c59be4a9f13cf8293e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5327906a7552497f89d780049b9abbf6" is_pre_election: true
I20260926 22:49:01.438117 15313 raft_consensus.cc:2454] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f00415f95be4c59be4a9f13cf8293e1 in term 0.
I20260926 22:49:01.438140 15173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022c1de9cdf94a25b98265888b0c201d" candidate_uuid: "6f00415f95be4c59be4a9f13cf8293e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df854172ad2a44ca9db32d1b411a7fc5" is_pre_election: true
I20260926 22:49:01.438226 15173 raft_consensus.cc:2454] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f00415f95be4c59be4a9f13cf8293e1 in term 0.
I20260926 22:49:01.438306 15384 leader_election.cc:302] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5327906a7552497f89d780049b9abbf6, 6f00415f95be4c59be4a9f13cf8293e1; no voters:
I20260926 22:49:01.438428 15525 raft_consensus.cc:2781] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:01.438472 15525 raft_consensus.cc:482] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:01.438491 15525 raft_consensus.cc:3037] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:01.439119 15525 raft_consensus.cc:504] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.439235 15525 leader_election.cc:288] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [CANDIDATE]: Term 1 election: Requested vote from peers 5327906a7552497f89d780049b9abbf6 (127.13.148.2:33209), df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127)
I20260926 22:49:01.439355 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022c1de9cdf94a25b98265888b0c201d" candidate_uuid: "6f00415f95be4c59be4a9f13cf8293e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5327906a7552497f89d780049b9abbf6"
I20260926 22:49:01.439424 15313 raft_consensus.cc:3037] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:01.439409 15173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022c1de9cdf94a25b98265888b0c201d" candidate_uuid: "6f00415f95be4c59be4a9f13cf8293e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df854172ad2a44ca9db32d1b411a7fc5"
I20260926 22:49:01.439481 15173 raft_consensus.cc:3037] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:01.440009 15313 raft_consensus.cc:2454] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f00415f95be4c59be4a9f13cf8293e1 in term 1.
I20260926 22:49:01.440091 15173 raft_consensus.cc:2454] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f00415f95be4c59be4a9f13cf8293e1 in term 1.
I20260926 22:49:01.440155 15384 leader_election.cc:302] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5327906a7552497f89d780049b9abbf6, 6f00415f95be4c59be4a9f13cf8293e1; no voters:
I20260926 22:49:01.440238 15525 raft_consensus.cc:2781] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:01.440335 15525 raft_consensus.cc:686] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 LEADER]: Becoming Leader. State: Replica: 6f00415f95be4c59be4a9f13cf8293e1, State: Running, Role: LEADER
I20260926 22:49:01.440394 15525 consensus_queue.cc:237] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } }
I20260926 22:49:01.440680 15562 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: TxnStatusTablet state changed. Reason: New leader 6f00415f95be4c59be4a9f13cf8293e1. Latest consensus state: current_term: 1 leader_uuid: "6f00415f95be4c59be4a9f13cf8293e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.440769 15562 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:01.440932 15564 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 22:49:01.441044 15564 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 22:49:01.441022 15015 catalog_manager.cc:5495] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f00415f95be4c59be4a9f13cf8293e1 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "6f00415f95be4c59be4a9f13cf8293e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:01.469480 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.466804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.13.148.3:39059)
0926 22:49:01.466957 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:01.466958 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:01.466965 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:01.466995 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:01.466995 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:01.467019 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:01.467079 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:01.467223 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.467723 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.468780 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.468980 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.469024 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.469195 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:01.469302 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:01.469350 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:01.469398 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 22:49:01.469413 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260926 22:49:01.469527 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.466564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:39059 (local address 127.0.0.1:38948)
0926 22:49:01.466637 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:01.466651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:01.466701 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:01.467103 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:01.467105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:01.467106 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:01.467159 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.467161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.467782 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.467783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.468707 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.468710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.469035 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.469036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.469102 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.469118 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:01.469351 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:01.469355 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:01.469396 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 22:49:01.469443 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 22:49:01.470603 15173 raft_consensus.cc:1261] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Refusing update from remote peer 6f00415f95be4c59be4a9f13cf8293e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:01.470723 15313 raft_consensus.cc:1261] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6 [term 1 FOLLOWER]: Refusing update from remote peer 6f00415f95be4c59be4a9f13cf8293e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:01.470875 15525 consensus_queue.cc:1035] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:01.470989 15562 consensus_queue.cc:1035] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:01.471524 15523 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: TxnStatusTablet state changed. Reason: New leader 6f00415f95be4c59be4a9f13cf8293e1. Latest consensus state: current_term: 1 leader_uuid: "6f00415f95be4c59be4a9f13cf8293e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.471616 15523 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:01.471621 15529 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: TxnStatusTablet state changed. Reason: New leader 6f00415f95be4c59be4a9f13cf8293e1. Latest consensus state: current_term: 1 leader_uuid: "6f00415f95be4c59be4a9f13cf8293e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.471696 15529 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:01.472280 15562 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f00415f95be4c59be4a9f13cf8293e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.472365 15562 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:01.472463 15523 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f00415f95be4c59be4a9f13cf8293e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.472532 15523 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P df854172ad2a44ca9db32d1b411a7fc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:01.473660 15525 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f00415f95be4c59be4a9f13cf8293e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.473766 15525 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P 6f00415f95be4c59be4a9f13cf8293e1: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:01.474117 15529 tablet_replica.cc:439] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f00415f95be4c59be4a9f13cf8293e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5327906a7552497f89d780049b9abbf6" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33209 } } peers { permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 } } peers { permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 } } }
I20260926 22:49:01.474195 15529 tablet_replica.cc:442] T 022c1de9cdf94a25b98265888b0c201d P 5327906a7552497f89d780049b9abbf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:01.478386 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.475925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:37419 (local address 127.13.148.3:39059)
0926 22:49:01.475964 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:01.475966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:01.476035 (+ 69us) server_negotiation.cc:407] Connection header received
0926 22:49:01.476116 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:01.476117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:01.476133 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:01.476170 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:01.476269 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.476703 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.477802 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.477989 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.478034 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.478189 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:01.478222 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:01.478255 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:01.478323 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 22:49:01.478338 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:01.478448 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.475850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:39059 (local address 127.13.148.2:37419)
0926 22:49:01.475978 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:01.475989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:01.476047 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:01.476181 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:01.476182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:01.476184 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:01.476228 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.476229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.476758 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.476759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.477697 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.477700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.478047 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.478048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.478109 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.478123 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:01.478267 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:01.478268 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:49:01.478320 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:49:01.478370 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":67,"threads_started":1}
I20260926 22:49:01.480444 15453 raft_consensus.cc:1261] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Refusing update from remote peer 5327906a7552497f89d780049b9abbf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:01.480510 15173 raft_consensus.cc:1261] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Refusing update from remote peer 5327906a7552497f89d780049b9abbf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:01.480628 15529 consensus_queue.cc:1035] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f00415f95be4c59be4a9f13cf8293e1" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:01.480737 15538 consensus_queue.cc:1035] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df854172ad2a44ca9db32d1b411a7fc5" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:01.481819 15571 mvcc.cc:205] Tried to move back new op lower bound from 7333736208302522368 to 7333736206855344128. Current Snapshot: MvccSnapshot[applied={T|T < 7333736208302522368}]
I20260926 22:49:01.492911 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.490090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33209 (local address 127.13.148.3:41829)
0926 22:49:01.490312 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:01.490322 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:01.490374 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:01.490480 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:01.490481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:01.490482 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:01.490540 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.490541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.491367 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.491367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.491957 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:01.491958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.492466 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.492467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:01.492536 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.492552 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:01.492774 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:01.492776 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:49:01.492815 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 22:49:01.492853 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260926 22:49:01.492923 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.490263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:41829 (local address 127.13.148.2:33209)
0926 22:49:01.490318 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:01.490320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:01.490362 (+ 42us) server_negotiation.cc:407] Connection header received
0926 22:49:01.490410 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:01.490411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:01.490423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:01.490479 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:01.490573 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.491307 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.492021 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.492347 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.492411 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.492656 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:01.492693 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:01.492751 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:01.492823 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 22:49:01.492847 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:01.500520 15015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:01.500631 15015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:01.501160 15015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b3538189e344eb0b7d12f728e62c5d9: Sending DeleteTablet for 3 replicas of tablet 7077f7bc971642bb96f41a38d40e1908
I20260926 22:49:01.501435 15284 tablet_service.cc:1514] Processing DeleteTablet for tablet 7077f7bc971642bb96f41a38d40e1908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:38014
I20260926 22:49:01.501652 15586 tablet_replica.cc:330] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: stopping tablet replica
I20260926 22:49:01.501705 15433 tablet_service.cc:1514] Processing DeleteTablet for tablet 7077f7bc971642bb96f41a38d40e1908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:38940
I20260926 22:49:01.501775 15586 raft_consensus.cc:2229] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:01.501835 15586 raft_consensus.cc:2258] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:01.501899 15153 tablet_service.cc:1514] Processing DeleteTablet for tablet 7077f7bc971642bb96f41a38d40e1908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:47464
I20260926 22:49:01.502002 15588 tablet_replica.cc:330] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: stopping tablet replica
I20260926 22:49:01.502101 15589 tablet_replica.cc:330] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: stopping tablet replica
I20260926 22:49:01.502122 15588 raft_consensus.cc:2229] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:01.502214 15589 raft_consensus.cc:2229] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:01.502223 15588 raft_consensus.cc:2258] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:01.502317 15589 raft_consensus.cc:2258] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:01.502494 15588 ts_tablet_manager.cc:1905] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:01.502502 15586 ts_tablet_manager.cc:1905] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:01.502586 15589 ts_tablet_manager.cc:1905] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:01.504469 15586 ts_tablet_manager.cc:1918] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:01.504488 15588 ts_tablet_manager.cc:1918] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:01.504525 15586 log.cc:1196] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/7077f7bc971642bb96f41a38d40e1908
I20260926 22:49:01.504539 15588 log.cc:1196] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/7077f7bc971642bb96f41a38d40e1908
I20260926 22:49:01.504730 15586 ts_tablet_manager.cc:1939] T 7077f7bc971642bb96f41a38d40e1908 P 5327906a7552497f89d780049b9abbf6: Deleting consensus metadata
I20260926 22:49:01.504748 15588 ts_tablet_manager.cc:1939] T 7077f7bc971642bb96f41a38d40e1908 P 6f00415f95be4c59be4a9f13cf8293e1: Deleting consensus metadata
I20260926 22:49:01.505009 15000 catalog_manager.cc:4841] TS 5327906a7552497f89d780049b9abbf6 (127.13.148.2:33209): tablet 7077f7bc971642bb96f41a38d40e1908 (table test-table [id=afd7f8e40f00484b955ef2e2ef24e515]) successfully deleted
I20260926 22:49:01.505066 15589 ts_tablet_manager.cc:1918] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:01.505119 15589 log.cc:1196] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/7077f7bc971642bb96f41a38d40e1908
I20260926 22:49:01.505326 15589 ts_tablet_manager.cc:1939] T 7077f7bc971642bb96f41a38d40e1908 P df854172ad2a44ca9db32d1b411a7fc5: Deleting consensus metadata
I20260926 22:49:01.505592 15001 catalog_manager.cc:4841] TS df854172ad2a44ca9db32d1b411a7fc5 (127.13.148.1:39127): tablet 7077f7bc971642bb96f41a38d40e1908 (table test-table [id=afd7f8e40f00484b955ef2e2ef24e515]) successfully deleted
I20260926 22:49:01.505676 15001 catalog_manager.cc:4841] TS 6f00415f95be4c59be4a9f13cf8293e1 (127.13.148.3:39059): tablet 7077f7bc971642bb96f41a38d40e1908 (table test-table [id=afd7f8e40f00484b955ef2e2ef24e515]) successfully deleted
Sep 26 22:49:01 dist-test-slave-98hb krb5kdc[14971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462940, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:49:01.512856 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.504342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.13.148.1:39127)
0926 22:49:01.504435 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:01.504437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:01.504445 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:01.504538 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:01.504539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:01.504550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:01.504578 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:01.505414 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.506029 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.506676 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.506964 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.507044 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.510106 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:01.510114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:01.510114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:01.510122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:01.511781 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:01.512080 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:01.512084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:01.512085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:01.512151 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:01.512315 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:01.512316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:01.512316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:01.512484 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:01.512621 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:01.512660 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 22:49:01.512739 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 22:49:01.513105 15211 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47470
I20260926 22:49:01.519582 15075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:01.514012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.13.148.62:46561)
0926 22:49:01.514054 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:01.514056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:01.514105 (+ 49us) server_negotiation.cc:407] Connection header received
0926 22:49:01.514172 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:01.514172 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:01.514183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:01.514238 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:01.514747 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.515227 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.515872 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:01.516072 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:01.516136 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:01.517178 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:01.517179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:01.517180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:01.517186 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:01.518587 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:01.518912 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:01.518913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:01.518913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:01.518966 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:01.519122 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:01.519123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:01.519123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:01.519248 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:01.519330 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:01.519425 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 22:49:01.519495 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 22:49:01.519852 15015 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51994
I20260926 22:49:01.520310 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 15088
I20260926 22:49:01.525707 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 15228
I20260926 22:49:01.530649 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 15368
I20260926 22:49:01.534703 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 14984
2026-09-26T22:49:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2495 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:01 dist-test-slave-98hb krb5kdc[15595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:01 dist-test-slave-98hb krb5kdc[15595](info): set up 2 sockets
Sep 26 22:49:01 dist-test-slave-98hb krb5kdc[15595](info): commencing operation
krb5kdc: starting...
W20260926 22:49:02.481366 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:02 dist-test-slave-98hb krb5kdc[15595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:02.589875 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46715
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:02.667922 15608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:02.668115 15608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:02.668135 15608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:02.669258 15608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:02.669301 15608 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 22:49:02.669313 15608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:02.669325 15608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:02.669348 15608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:02.669363 15608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:02.670833 15608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:02.671049 15608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:02.671267 15608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:02.675758 15613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:02.675799 15616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:02.675818 15614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:02.675915 15608 server_base.cc:1034] running on GCE node
I20260926 22:49:02.676177 15608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:02.676452 15608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:02.677591 15608 hybrid_clock.cc:648] HybridClock initialized: now 1790462942677580 us; error 24 us; skew 500 ppm
Sep 26 22:49:02 dist-test-slave-98hb krb5kdc[15595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462942, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:02.679917 15608 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:02.680343 15608 webserver.cc:460] Webserver started at http://127.13.148.62:32827/ using document root <none> and password file <none>
I20260926 22:49:02.680532 15608 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:02.680578 15608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:02.680698 15608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:02.681700 15608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "181d8e2ecedc44608d282a19351efae1"
format_stamp: "Formatted at 2026-09-26 22:49:02 on dist-test-slave-98hb"
server_key: "fe624d70715d5638e095162f33bfc838"
server_key_iv: "398b42f15982465903ee1f05b58ca247"
server_key_version: "encryptionkey@0"
I20260926 22:49:02.682075 15608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "181d8e2ecedc44608d282a19351efae1"
format_stamp: "Formatted at 2026-09-26 22:49:02 on dist-test-slave-98hb"
server_key: "fe624d70715d5638e095162f33bfc838"
server_key_iv: "398b42f15982465903ee1f05b58ca247"
server_key_version: "encryptionkey@0"
I20260926 22:49:02.683290 15608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:02.683969 15623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:02.684096 15608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:02.684176 15608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "181d8e2ecedc44608d282a19351efae1"
format_stamp: "Formatted at 2026-09-26 22:49:02 on dist-test-slave-98hb"
server_key: "fe624d70715d5638e095162f33bfc838"
server_key_iv: "398b42f15982465903ee1f05b58ca247"
server_key_version: "encryptionkey@0"
I20260926 22:49:02.684242 15608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:02.705454 15608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:02.707458 15608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:02.707602 15608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:02.711890 15608 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46715
I20260926 22:49:02.711959 15685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46715 every 8 connection(s)
I20260926 22:49:02.712261 15608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:02.712849 15686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:02.713877 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 15608
I20260926 22:49:02.713949 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:02.714092 13904 external_mini_cluster.cc:1428] Setting key d448675a5b777c12cabf3c051995e212
I20260926 22:49:02.715303 15686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: Bootstrap starting.
I20260926 22:49:02.715857 15686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:02.716109 15686 log.cc:824] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:02.716769 15686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: No bootstrap required, opened a new log
I20260926 22:49:02.718519 15686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181d8e2ecedc44608d282a19351efae1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46715 } }
I20260926 22:49:02.718695 15686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:02.718770 15686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181d8e2ecedc44608d282a19351efae1, State: Initialized, Role: FOLLOWER
I20260926 22:49:02.718906 15686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181d8e2ecedc44608d282a19351efae1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46715 } }
I20260926 22:49:02.718982 15686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:02.719023 15686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:02.719070 15686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:02.719560 15686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181d8e2ecedc44608d282a19351efae1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46715 } }
I20260926 22:49:02.719656 15686 leader_election.cc:302] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181d8e2ecedc44608d282a19351efae1; no voters:
I20260926 22:49:02.719816 15686 leader_election.cc:288] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:02.719847 15691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:02.720009 15691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [term 1 LEADER]: Becoming Leader. State: Replica: 181d8e2ecedc44608d282a19351efae1, State: Running, Role: LEADER
I20260926 22:49:02.720057 15686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 22:49:02 dist-test-slave-98hb krb5kdc[15595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:02.720167 15691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181d8e2ecedc44608d282a19351efae1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46715 } }
I20260926 22:49:02.720498 15691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181d8e2ecedc44608d282a19351efae1. Latest consensus state: current_term: 1 leader_uuid: "181d8e2ecedc44608d282a19351efae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181d8e2ecedc44608d282a19351efae1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46715 } } }
I20260926 22:49:02.720546 15693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181d8e2ecedc44608d282a19351efae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181d8e2ecedc44608d282a19351efae1" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46715 } } }
I20260926 22:49:02.720566 15691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:02.720592 15693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:02.721176 15697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:02.721760 15697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:02.723145 15697 catalog_manager.cc:1347] Generated new cluster ID: 1cff32bd79964406b6fafd62a464cf49
I20260926 22:49:02.723201 15697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:02.724733 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:02.714542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.13.148.62:46715)
0926 22:49:02.714711 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:02.714722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:02.714747 (+ 25us) server_negotiation.cc:407] Connection header received
0926 22:49:02.715148 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:02.715154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:02.715200 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:02.715335 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:02.715929 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.717020 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.717779 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.718184 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.718284 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:02.720745 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:02.720749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:02.720749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:02.720762 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:02.722712 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:02.723148 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:02.723151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:02.723152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:02.723282 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:02.723540 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:02.723541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:02.723541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:02.723835 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:02.723959 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:02.724384 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 22:49:02.724520 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260926 22:49:02.734046 15697 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:02.734515 15697 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:02.745587 15697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: Generated new TSK 0
I20260926 22:49:02.745816 15697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:02.774011 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46715
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:02.853050 15712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:02.853257 15712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:02.853281 15712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:02.854400 15712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:02.854439 15712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:02.854458 15712 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 22:49:02.854481 15712 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:02.854511 15712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:02.855914 15712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:02.856179 15712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:02.856389 15712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:02.861104 15718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:02.861120 15720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:02.861200 15712 server_base.cc:1034] running on GCE node
W20260926 22:49:02.861121 15717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:02.861490 15712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:02.861754 15712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:02.862923 15712 hybrid_clock.cc:648] HybridClock initialized: now 1790462942862911 us; error 26 us; skew 500 ppm
Sep 26 22:49:02 dist-test-slave-98hb krb5kdc[15595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462942, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:02.865271 15712 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:02.865662 15712 webserver.cc:460] Webserver started at http://127.13.148.1:42115/ using document root <none> and password file <none>
I20260926 22:49:02.865895 15712 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:02.865940 15712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:02.866055 15712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:02.867066 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "1781b828b532420f9017d00ab6a01577"
format_stamp: "Formatted at 2026-09-26 22:49:02 on dist-test-slave-98hb"
server_key: "b9c04dfb92354feecafc1b6806e76de2"
server_key_iv: "0ec28d8b94637cb7208661d94c4cd169"
server_key_version: "encryptionkey@0"
I20260926 22:49:02.867380 15712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "1781b828b532420f9017d00ab6a01577"
format_stamp: "Formatted at 2026-09-26 22:49:02 on dist-test-slave-98hb"
server_key: "b9c04dfb92354feecafc1b6806e76de2"
server_key_iv: "0ec28d8b94637cb7208661d94c4cd169"
server_key_version: "encryptionkey@0"
I20260926 22:49:02.868613 15712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:02.869221 15727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:02.869374 15712 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:02.869443 15712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "1781b828b532420f9017d00ab6a01577"
format_stamp: "Formatted at 2026-09-26 22:49:02 on dist-test-slave-98hb"
server_key: "b9c04dfb92354feecafc1b6806e76de2"
server_key_iv: "0ec28d8b94637cb7208661d94c4cd169"
server_key_version: "encryptionkey@0"
I20260926 22:49:02.869503 15712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:02.887010 15712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:02.889122 15712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:02.889243 15712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:02.889726 15712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:02.889799 15712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:02.889827 15712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:02.889848 15712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:49:02 dist-test-slave-98hb krb5kdc[15595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462942, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:02.896005 15712 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:45121
I20260926 22:49:02.896075 15842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:45121 every 8 connection(s)
I20260926 22:49:02.896386 15712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:02.898505 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 15712
I20260926 22:49:02.898602 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:02.898792 13904 external_mini_cluster.cc:1428] Setting key 93ea67d1b81f65c4e0d631422ccd47c8
I20260926 22:49:02.898983 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:02.890199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:47721 (local address 127.13.148.62:46715)
0926 22:49:02.890247 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:02.890250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:02.890463 (+ 213us) server_negotiation.cc:407] Connection header received
0926 22:49:02.890938 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:02.890939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:02.890952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:02.891024 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:02.891800 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.892523 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.893985 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.894285 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.894366 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:02.896448 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:02.896449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:02.896450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:02.896457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:02.897824 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:02.898145 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:02.898145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:02.898145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:02.898188 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:02.898357 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:02.898358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:02.898358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:02.898449 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:02.898492 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:02.898816 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 22:49:02.898895 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:49:02.899147 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:02.890174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46715 (local address 127.13.148.1:47721)
0926 22:49:02.890341 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:02.890381 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:02.890854 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:02.891100 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:02.891104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:02.891409 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:02.891685 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:02.891692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.892615 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.892616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:02.893904 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:02.893907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.894366 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.894366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:02.894460 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:02.894938 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:02.894947 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:02.896364 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:02.897876 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:02.897881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:02.897883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:02.898086 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:02.898222 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:02.898222 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:02.898223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:02.898307 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:02.898501 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:02.898506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:02.898665 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 22:49:02.898789 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 22:49:02.898950 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260926 22:49:02.899701 15843 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46715
I20260926 22:49:02.899783 15843 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:02.899961 15843 heartbeater.cc:511] Master 127.13.148.62:46715 requested a full tablet report, sending...
I20260926 22:49:02.900492 15640 ts_manager.cc:194] Registered new tserver with Master: 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121)
I20260926 22:49:02.901355 15640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:47721
I20260926 22:49:02.905988 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:02.900845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46715 (local address 127.13.148.1:40829)
0926 22:49:02.900965 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:02.900974 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:02.901027 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:02.901154 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:02.901154 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:02.901320 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:02.901365 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:02.901367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.902109 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.902110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:02.902620 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:02.902621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.902945 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.902945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:02.903016 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:02.903453 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:02.903456 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:02.903715 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:02.905044 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:02.905045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:02.905045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:02.905200 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:02.905386 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:02.905387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:02.905387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:02.905443 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:02.905647 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:02.905648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:02.905695 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 22:49:02.905790 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 22:49:02.905889 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260926 22:49:02.905983 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:02.900802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:40829 (local address 127.13.148.62:46715)
0926 22:49:02.900842 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:02.900843 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:02.901026 (+ 183us) server_negotiation.cc:407] Connection header received
0926 22:49:02.901080 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:02.901081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:02.901090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:02.901139 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:02.901430 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.902024 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.902688 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:02.902888 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:02.902942 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:02.903790 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:02.903792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:02.903792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:02.903799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:02.904972 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:02.905280 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:02.905281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:02.905282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:02.905335 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:02.905490 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:02.905491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:02.905491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:02.905596 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:02.905650 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:02.905817 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 22:49:02.905889 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:02.923894 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46715
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:03.001420 15852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:03.001616 15852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:03.001638 15852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:03.002755 15852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:03.002801 15852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:03.002825 15852 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 22:49:03.002846 15852 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:03.002868 15852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:03.004272 15852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:03.004470 15852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:03.004683 15852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:03.009368 15858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:03.009356 15857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:03.009359 15860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:03.009385 15852 server_base.cc:1034] running on GCE node
I20260926 22:49:03.009716 15852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:03.009964 15852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:03.011101 15852 hybrid_clock.cc:648] HybridClock initialized: now 1790462943011088 us; error 23 us; skew 500 ppm
Sep 26 22:49:03 dist-test-slave-98hb krb5kdc[15595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:03.013367 15852 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:03.013788 15852 webserver.cc:460] Webserver started at http://127.13.148.2:42977/ using document root <none> and password file <none>
I20260926 22:49:03.013983 15852 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:03.014024 15852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:03.014148 15852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:03.015120 15852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "a0d44dafd82443baa1c13f91ebd2f01e"
format_stamp: "Formatted at 2026-09-26 22:49:03 on dist-test-slave-98hb"
server_key: "758cf52289387c74bed484baaf4d957d"
server_key_iv: "82a3eb827a0f0147b5915c699f8876dc"
server_key_version: "encryptionkey@0"
I20260926 22:49:03.015445 15852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "a0d44dafd82443baa1c13f91ebd2f01e"
format_stamp: "Formatted at 2026-09-26 22:49:03 on dist-test-slave-98hb"
server_key: "758cf52289387c74bed484baaf4d957d"
server_key_iv: "82a3eb827a0f0147b5915c699f8876dc"
server_key_version: "encryptionkey@0"
I20260926 22:49:03.016597 15852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:49:03.017284 15867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.017419 15852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.017500 15852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "a0d44dafd82443baa1c13f91ebd2f01e"
format_stamp: "Formatted at 2026-09-26 22:49:03 on dist-test-slave-98hb"
server_key: "758cf52289387c74bed484baaf4d957d"
server_key_iv: "82a3eb827a0f0147b5915c699f8876dc"
server_key_version: "encryptionkey@0"
I20260926 22:49:03.017565 15852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:03.031534 15852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:03.033124 15852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:03.033257 15852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:03.033720 15852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:03.033828 15852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.033849 15852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:03.033860 15852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 22:49:03 dist-test-slave-98hb krb5kdc[15595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:03.039973 15852 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:33593
I20260926 22:49:03.040064 15982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:33593 every 8 connection(s)
I20260926 22:49:03.040364 15852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:03.042603 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.034151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:49983 (local address 127.13.148.62:46715)
0926 22:49:03.034208 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.034211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.034383 (+ 172us) server_negotiation.cc:407] Connection header received
0926 22:49:03.034870 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.034871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.034883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.034930 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:03.035535 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.036282 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.037705 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.038046 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.038133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.040373 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:03.040374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:03.040375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:03.040382 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:03.041554 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.041955 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.041956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.041957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.042003 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.042159 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.042159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.042159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.042246 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:03.042289 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.042477 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.042541 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:49:03.042758 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.034128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46715 (local address 127.13.148.2:49983)
0926 22:49:03.034266 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.034312 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.034774 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.034997 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.035003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.035267 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:03.035449 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.035454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.036384 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.036385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.037607 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.037611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.038141 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.038142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.038257 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.038750 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:03.038760 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:03.040284 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:03.041617 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.041622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.041623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.041897 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.042041 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.042042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.042042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.042126 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.042286 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:03.042289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:03.042392 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.042467 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.042580 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260926 22:49:03.043208 15983 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46715
I20260926 22:49:03.043299 15983 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:03.043447 15983 heartbeater.cc:511] Master 127.13.148.62:46715 requested a full tablet report, sending...
I20260926 22:49:03.043834 15640 ts_manager.cc:194] Registered new tserver with Master: a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593)
I20260926 22:49:03.044267 15640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:49983
I20260926 22:49:03.048251 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 15852
I20260926 22:49:03.048337 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:03.048527 13904 external_mini_cluster.cc:1428] Setting key 5fa6df08a312565e94feae908567bf57
I20260926 22:49:03.049408 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.044160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:50543 (local address 127.13.148.62:46715)
0926 22:49:03.044200 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.044202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.044352 (+ 150us) server_negotiation.cc:407] Connection header received
0926 22:49:03.044397 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.044398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.044407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.044451 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:03.044763 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.045309 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.045976 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.046185 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.046230 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.046892 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:03.046894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:03.046894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:03.046899 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:03.048074 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.048680 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.048682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.048682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.048748 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.048933 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.048933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.048934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.049093 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:03.049159 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.049295 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.049351 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:03.049463 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.044135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46715 (local address 127.13.148.2:50543)
0926 22:49:03.044275 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.044286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.044351 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.044476 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.044477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.044647 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:03.044692 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.044694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.045382 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.045383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.045918 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.045920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.046237 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.046237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.046295 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.046595 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:03.046599 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:03.046831 (+ 232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:03.048431 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.048432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.048432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.048617 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.048813 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.048813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.048814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.048877 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.049165 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:03.049166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:03.049209 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.049276 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.049381 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:03.072022 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46715
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:03.150955 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:03.151152 15992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:03.151183 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:03.152292 15992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:03.152331 15992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:03.152357 15992 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 22:49:03.152371 15992 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:03.152395 15992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:03.153854 15992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:46715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:03.154076 15992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:03.154290 15992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:03.158946 16000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:03.158946 15998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:03.159034 15992 server_base.cc:1034] running on GCE node
W20260926 22:49:03.158948 15997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:03.159410 15992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:03.159644 15992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:03.160776 15992 hybrid_clock.cc:648] HybridClock initialized: now 1790462943160766 us; error 33 us; skew 500 ppm
Sep 26 22:49:03 dist-test-slave-98hb krb5kdc[15595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:03.163012 15992 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:03.163408 15992 webserver.cc:460] Webserver started at http://127.13.148.3:44617/ using document root <none> and password file <none>
I20260926 22:49:03.163595 15992 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:03.163641 15992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:03.163762 15992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:03.164762 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd"
format_stamp: "Formatted at 2026-09-26 22:49:03 on dist-test-slave-98hb"
server_key: "96529713ec7d81e250491c8c46d0b49b"
server_key_iv: "14f8b89583dc6cf9212b5839f8cf0608"
server_key_version: "encryptionkey@0"
I20260926 22:49:03.165076 15992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd"
format_stamp: "Formatted at 2026-09-26 22:49:03 on dist-test-slave-98hb"
server_key: "96529713ec7d81e250491c8c46d0b49b"
server_key_iv: "14f8b89583dc6cf9212b5839f8cf0608"
server_key_version: "encryptionkey@0"
I20260926 22:49:03.166345 15992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 22:49:03.167012 16007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.167173 15992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:03.167249 15992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd"
format_stamp: "Formatted at 2026-09-26 22:49:03 on dist-test-slave-98hb"
server_key: "96529713ec7d81e250491c8c46d0b49b"
server_key_iv: "14f8b89583dc6cf9212b5839f8cf0608"
server_key_version: "encryptionkey@0"
I20260926 22:49:03.167317 15992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:03.198055 15992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:03.199877 15992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:03.200006 15992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:03.200510 15992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:03.200551 15992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.200584 15992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:03.200608 15992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.206861 15992 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36681
I20260926 22:49:03.206890 16122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36681 every 8 connection(s)
Sep 26 22:49:03 dist-test-slave-98hb krb5kdc[15595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462943, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:03.207231 15992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:03.209870 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.201007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58927 (local address 127.13.148.62:46715)
0926 22:49:03.201053 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.201055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.201297 (+ 242us) server_negotiation.cc:407] Connection header received
0926 22:49:03.201856 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.201858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.201872 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.201946 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:03.202547 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.203291 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.204738 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.205033 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.205109 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.207519 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:03.207520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:03.207521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:03.207529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:03.208825 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.209152 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.209153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.209153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.209196 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.209379 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.209380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.209380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.209471 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:03.209513 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.209701 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.209809 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:49:03.210016 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.201016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46715 (local address 127.13.148.3:58927)
0926 22:49:03.201167 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.201203 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.201710 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.202022 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.202026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.202281 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:03.202474 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.202479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.203381 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.203382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.204633 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.204636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.205116 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.205117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.205254 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.205709 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:03.205718 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:03.207406 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:03.208891 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.208894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.208896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.209096 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.209242 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.209243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.209243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.209337 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.209512 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:03.209514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:03.209617 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.209690 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.209845 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260926 22:49:03.210462 16123 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46715
I20260926 22:49:03.210536 16123 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:03.210693 16123 heartbeater.cc:511] Master 127.13.148.62:46715 requested a full tablet report, sending...
I20260926 22:49:03.211071 15639 ts_manager.cc:194] Registered new tserver with Master: bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:03.211524 15639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:58927
I20260926 22:49:03.216143 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.211405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:36121 (local address 127.13.148.62:46715)
0926 22:49:03.211438 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.211440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.211533 (+ 93us) server_negotiation.cc:407] Connection header received
0926 22:49:03.211587 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.211587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.211596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.211651 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:03.211961 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.212417 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.213065 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.213258 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.213308 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.214076 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:03.214077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:03.214077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:03.214083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:03.215203 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.215484 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.215485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.215486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.215535 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.215687 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.215687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.215688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.215792 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:03.215844 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.215973 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.216037 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 22:49:03.216149 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.211362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46715 (local address 127.13.148.3:36121)
0926 22:49:03.211478 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.211487 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.211536 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.211704 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.211705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.211851 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:03.211895 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.211896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.212500 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.212501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.212998 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.212999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.213312 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.213313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.213384 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.213723 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:03.213742 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:03.214001 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:03.215276 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.215277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.215277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.215427 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.215584 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.215585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.215585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.215639 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.215842 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:03.215842 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:03.215885 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.215945 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.216039 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260926 22:49:03.216539 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 15992
I20260926 22:49:03.216650 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:03.216818 13904 external_mini_cluster.cc:1428] Setting key bc78bd39c657abc87a6336a66cfa9eb1
I20260926 22:49:03.217589 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 22:49:03 dist-test-slave-98hb krb5kdc[15595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:49:03 dist-test-slave-98hb krb5kdc[15595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:03.237399 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.231804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.13.148.62:46715)
0926 22:49:03.231858 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.231860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.231882 (+ 22us) server_negotiation.cc:407] Connection header received
0926 22:49:03.231902 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.231903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.231912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.231950 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:03.232402 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.232872 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.233497 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.233660 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.233699 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.235612 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:03.235613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:03.235614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:03.235620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:03.236634 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.236904 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.236905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.236905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.236946 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.237062 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.237062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.237062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.237148 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:03.237225 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.237283 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.237336 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:49:03.238785 15639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:03.240247 15639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:03.246304 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.242189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45121 (local address 127.0.0.1:42808)
0926 22:49:03.242550 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.242557 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.242596 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.242741 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.242743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.242745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:03.242805 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.242807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.244166 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.244167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.245082 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.245085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.245902 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.245904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.245981 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.246097 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.246169 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.246221 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":36,"thread_start_us":48,"threads_started":1}
I20260926 22:49:03.246330 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.242358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.13.148.1:45121)
0926 22:49:03.242526 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.242531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.242592 (+ 61us) server_negotiation.cc:407] Connection header received
0926 22:49:03.242652 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.242654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.242669 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.242735 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:03.242934 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.244075 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.245164 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.245832 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.245902 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.245960 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.246192 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.246233 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260926 22:49:03.246701 16138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.242189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36681 (local address 127.0.0.1:40438)
0926 22:49:03.242359 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.242397 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.242459 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.242851 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.242852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.242854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:03.242892 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.242893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.244357 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.244358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.245402 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.245405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.246311 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.246312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.246372 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.246376 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.246593 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.246634 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260926 22:49:03.246788 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.242193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33593 (local address 127.0.0.1:43242)
0926 22:49:03.243014 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.243023 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.243114 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.243276 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.243277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.243278 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:03.243314 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.243316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.244170 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.244172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.245615 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.245617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.246547 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.246548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.246605 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.246610 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.246653 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.246691 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":46,"thread_start_us":75,"threads_started":1}
I20260926 22:49:03.247004 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.242407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.13.148.3:36681)
0926 22:49:03.242721 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.242724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.242733 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:03.242769 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.242771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.242783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.242854 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:03.243200 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.244287 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.245483 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.246150 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.246221 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.246274 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.246873 (+ 599us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.246914 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":79,"threads_started":1}
I20260926 22:49:03.247062 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.242551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.13.148.2:33593)
0926 22:49:03.242685 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.242690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.243085 (+ 395us) server_negotiation.cc:407] Connection header received
0926 22:49:03.243189 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.243191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.243206 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.243269 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:03.243363 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.244109 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.245679 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.246399 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.246463 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.246516 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.246677 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.246710 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260926 22:49:03.247123 15777 tablet_service.cc:1467] Processing CreateTablet for tablet bcb4eb4d5c7e469eb0f698d4038c2e70 (DEFAULT_TABLE table=test-table [id=358dcc95d56044aebe012c32bee37357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:03.247468 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb4eb4d5c7e469eb0f698d4038c2e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.250203 16143 tablet_bootstrap.cc:492] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: Bootstrap starting.
I20260926 22:49:03.250186 16057 tablet_service.cc:1467] Processing CreateTablet for tablet bcb4eb4d5c7e469eb0f698d4038c2e70 (DEFAULT_TABLE table=test-table [id=358dcc95d56044aebe012c32bee37357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:03.250535 16057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb4eb4d5c7e469eb0f698d4038c2e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.250771 16143 tablet_bootstrap.cc:654] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.251029 16143 log.cc:824] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:03.251632 16143 tablet_bootstrap.cc:492] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: No bootstrap required, opened a new log
I20260926 22:49:03.251684 16143 ts_tablet_manager.cc:1397] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:03.252841 16145 tablet_bootstrap.cc:492] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: Bootstrap starting.
I20260926 22:49:03.253228 15917 tablet_service.cc:1467] Processing CreateTablet for tablet bcb4eb4d5c7e469eb0f698d4038c2e70 (DEFAULT_TABLE table=test-table [id=358dcc95d56044aebe012c32bee37357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:03.253348 16143 raft_consensus.cc:348] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.253511 16143 raft_consensus.cc:374] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.253540 16143 raft_consensus.cc:729] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1781b828b532420f9017d00ab6a01577, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.253554 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb4eb4d5c7e469eb0f698d4038c2e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.253666 16143 consensus_queue.cc:260] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.253782 16145 tablet_bootstrap.cc:654] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.253949 16143 ts_tablet_manager.cc:1428] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:03.254087 16145 log.cc:824] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:03.254684 16145 tablet_bootstrap.cc:492] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: No bootstrap required, opened a new log
I20260926 22:49:03.254742 16145 ts_tablet_manager.cc:1397] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:03.256017 16145 raft_consensus.cc:348] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.256151 15843 heartbeater.cc:503] Master 127.13.148.62:46715 was elected leader, sending a full tablet report...
I20260926 22:49:03.256168 16145 raft_consensus.cc:374] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.256217 16145 raft_consensus.cc:729] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fb964b20e46ccab6b52cdefc5b6dd, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.256111 16148 tablet_bootstrap.cc:492] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Bootstrap starting.
I20260926 22:49:03.256331 16145 consensus_queue.cc:260] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.256572 16145 ts_tablet_manager.cc:1428] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:03.256810 16148 tablet_bootstrap.cc:654] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.256831 16123 heartbeater.cc:503] Master 127.13.148.62:46715 was elected leader, sending a full tablet report...
I20260926 22:49:03.257112 16148 log.cc:824] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:03.257825 16148 tablet_bootstrap.cc:492] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: No bootstrap required, opened a new log
I20260926 22:49:03.257889 16148 ts_tablet_manager.cc:1397] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:03.259132 16148 raft_consensus.cc:348] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.259246 16148 raft_consensus.cc:374] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.259264 16148 raft_consensus.cc:729] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d44dafd82443baa1c13f91ebd2f01e, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.259344 16148 consensus_queue.cc:260] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.259521 16148 ts_tablet_manager.cc:1428] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:03.259562 15983 heartbeater.cc:503] Master 127.13.148.62:46715 was elected leader, sending a full tablet report...
W20260926 22:49:03.291182 15984 tablet.cc:2369] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:03.332144 16146 raft_consensus.cc:482] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:03.332273 16146 raft_consensus.cc:504] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.332654 16146 leader_election.cc:288] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:03.336428 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.333019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42969 (local address 127.13.148.2:33593)
0926 22:49:03.333074 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.333076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.333083 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:03.333111 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.333111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.333119 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.333173 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:03.333320 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.334070 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.335487 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.336039 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.336141 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.336150 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.336311 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.336351 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:49:03.336423 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.332935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:50705 (local address 127.13.148.3:36681)
0926 22:49:03.332977 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.332979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.333179 (+ 200us) server_negotiation.cc:407] Connection header received
0926 22:49:03.333229 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.333230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.333239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.333284 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:03.333421 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.334145 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.335486 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.336049 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.336141 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.336150 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.336311 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.336349 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:49:03.336452 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.332829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33593 (local address 127.13.148.1:42969)
0926 22:49:03.332878 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.332889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.332927 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.333187 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.333188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.333189 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:03.333258 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.333259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.334143 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.334144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.335377 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.335380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.336142 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.336143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.336245 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.336247 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.336305 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.336352 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":6}
I20260926 22:49:03.336459 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.332831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36681 (local address 127.13.148.1:50705)
0926 22:49:03.333109 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.333118 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.333174 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.333302 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.333303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.333305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:03.333366 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.333367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.334210 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.334210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.335377 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.335380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.336145 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.336145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.336238 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.336242 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.336307 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.336352 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":191,"thread_start_us":95,"threads_started":1}
I20260926 22:49:03.336843 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb4eb4d5c7e469eb0f698d4038c2e70" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" is_pre_election: true
I20260926 22:49:03.336841 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb4eb4d5c7e469eb0f698d4038c2e70" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" is_pre_election: true
I20260926 22:49:03.336980 15937 raft_consensus.cc:2454] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 0.
I20260926 22:49:03.336980 16071 raft_consensus.cc:2454] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 0.
I20260926 22:49:03.337195 15730 leader_election.cc:302] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1781b828b532420f9017d00ab6a01577, bf9fb964b20e46ccab6b52cdefc5b6dd; no voters:
I20260926 22:49:03.337363 16146 raft_consensus.cc:2781] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:03.337414 16146 raft_consensus.cc:482] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:03.337430 16146 raft_consensus.cc:3037] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.338121 16146 raft_consensus.cc:504] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.338232 16146 leader_election.cc:288] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 election: Requested vote from peers a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:03.338382 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb4eb4d5c7e469eb0f698d4038c2e70" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d44dafd82443baa1c13f91ebd2f01e"
I20260926 22:49:03.338397 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb4eb4d5c7e469eb0f698d4038c2e70" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd"
I20260926 22:49:03.338454 15937 raft_consensus.cc:3037] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.338456 16071 raft_consensus.cc:3037] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.339084 16071 raft_consensus.cc:2454] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 1.
I20260926 22:49:03.339125 15937 raft_consensus.cc:2454] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 1.
I20260926 22:49:03.339243 15730 leader_election.cc:302] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1781b828b532420f9017d00ab6a01577, bf9fb964b20e46ccab6b52cdefc5b6dd; no voters:
I20260926 22:49:03.339370 16146 raft_consensus.cc:2781] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:03.339510 16146 raft_consensus.cc:686] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 1 LEADER]: Becoming Leader. State: Replica: 1781b828b532420f9017d00ab6a01577, State: Running, Role: LEADER
I20260926 22:49:03.339597 16146 consensus_queue.cc:237] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.340358 15639 catalog_manager.cc:5495] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1781b828b532420f9017d00ab6a01577 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "1781b828b532420f9017d00ab6a01577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } health_report { overall_health: UNKNOWN } } }
Sep 26 22:49:03 dist-test-slave-98hb krb5kdc[15595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462942, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:03.388510 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.382421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46715 (local address 127.0.0.1:57290)
0926 22:49:03.382554 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.382564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.382635 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.382753 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.382753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.383062 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:03.383108 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.383110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.383754 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.383756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.384265 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.384267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.384560 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.384560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.384605 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.385044 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:03.385048 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:03.386322 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:03.387696 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.387701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.387703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.387881 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.388020 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:03.388021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:03.388021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:03.388067 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:03.388240 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:03.388244 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:03.388283 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.388332 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.388415 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260926 22:49:03.388551 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.382471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57290 (local address 127.13.148.62:46715)
0926 22:49:03.382525 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.382527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.382629 (+ 102us) server_negotiation.cc:407] Connection header received
0926 22:49:03.382694 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.382695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.382703 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.382744 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:03.383188 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.383674 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.384340 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.384506 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.384557 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.386391 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:03.386393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:03.386393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:03.386399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:03.387634 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.387936 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.387936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.387937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.387979 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:03.388100 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:03.388100 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:03.388101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:03.388187 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:03.388246 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.388350 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.388455 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":30}
I20260926 22:49:03.390326 15639 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:57290:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 22:49:03.390509 15639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:03.391768 15777 tablet_service.cc:1467] Processing CreateTablet for tablet 733898382f1844f6b43db71f1ce7906c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0369f6394de49be8b20a7d72a7bede4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:03.391804 16057 tablet_service.cc:1467] Processing CreateTablet for tablet 733898382f1844f6b43db71f1ce7906c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0369f6394de49be8b20a7d72a7bede4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:03.391942 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733898382f1844f6b43db71f1ce7906c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.391942 16057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733898382f1844f6b43db71f1ce7906c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.393468 16145 tablet_bootstrap.cc:492] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: Bootstrap starting.
I20260926 22:49:03.393481 16143 tablet_bootstrap.cc:492] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: Bootstrap starting.
I20260926 22:49:03.393600 15917 tablet_service.cc:1467] Processing CreateTablet for tablet 733898382f1844f6b43db71f1ce7906c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0369f6394de49be8b20a7d72a7bede4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:03.393716 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 733898382f1844f6b43db71f1ce7906c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.393930 16143 tablet_bootstrap.cc:654] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.394001 16145 tablet_bootstrap.cc:654] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.394546 16143 tablet_bootstrap.cc:492] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: No bootstrap required, opened a new log
I20260926 22:49:03.394588 16143 ts_tablet_manager.cc:1397] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:03.394610 16145 tablet_bootstrap.cc:492] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: No bootstrap required, opened a new log
I20260926 22:49:03.394667 16145 ts_tablet_manager.cc:1397] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:03.394721 16143 raft_consensus.cc:348] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.394843 16143 raft_consensus.cc:374] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.394871 16143 raft_consensus.cc:729] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1781b828b532420f9017d00ab6a01577, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.394827 16145 raft_consensus.cc:348] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.394891 16145 raft_consensus.cc:374] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.394915 16145 raft_consensus.cc:729] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fb964b20e46ccab6b52cdefc5b6dd, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.394901 16143 consensus_queue.cc:260] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.394954 16148 tablet_bootstrap.cc:492] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: Bootstrap starting.
I20260926 22:49:03.394951 16145 consensus_queue.cc:260] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.394991 16143 ts_tablet_manager.cc:1428] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:03.395027 16145 ts_tablet_manager.cc:1428] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:03.395094 16146 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.395156 16146 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:03.395195 16149 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.395279 16149 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:03.395603 16148 tablet_bootstrap.cc:654] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.396242 16148 tablet_bootstrap.cc:492] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: No bootstrap required, opened a new log
I20260926 22:49:03.396282 16148 ts_tablet_manager.cc:1397] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:03.396386 16148 raft_consensus.cc:348] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.396436 16148 raft_consensus.cc:374] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.396447 16148 raft_consensus.cc:729] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d44dafd82443baa1c13f91ebd2f01e, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.396474 16148 consensus_queue.cc:260] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.396553 16148 ts_tablet_manager.cc:1428] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.396631 16151 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.396709 16151 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 22:49:03.397037 15844 tablet.cc:2369] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:49:03.457823 16124 tablet.cc:2369] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:03.597858 16151 raft_consensus.cc:482] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:03.597958 16151 raft_consensus.cc:504] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.598261 16151 leader_election.cc:288] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:03.601052 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.598508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39337 (local address 127.13.148.1:45121)
0926 22:49:03.598555 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.598557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.598636 (+ 79us) server_negotiation.cc:407] Connection header received
0926 22:49:03.598692 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.598693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.598702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.598743 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:03.598877 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.599390 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.600398 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.600754 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.600808 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.600818 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.600945 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.600982 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:49:03.601066 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.598546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:48715 (local address 127.13.148.3:36681)
0926 22:49:03.598584 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.598586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.598623 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:49:03.598702 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.598704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.598726 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.598777 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:03.598924 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.599399 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.600390 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.600749 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.600831 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.600840 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.600969 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.601002 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:03.601094 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.598431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45121 (local address 127.13.148.2:39337)
0926 22:49:03.598564 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.598577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.598639 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.598758 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.598759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.598761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:03.598819 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.598822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.599456 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.599457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.600322 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.600325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.600816 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.600817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.600886 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.600890 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.600932 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.600980 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260926 22:49:03.601096 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.598432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36681 (local address 127.13.148.2:48715)
0926 22:49:03.598570 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.598579 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.598638 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.598799 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.598800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.598802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:03.598865 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.598867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.599467 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.599468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.600315 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.600317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.600831 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.600832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.600895 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.600898 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.600944 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.600983 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":40,"thread_start_us":65,"threads_started":1}
I20260926 22:49:03.601395 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733898382f1844f6b43db71f1ce7906c" candidate_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1781b828b532420f9017d00ab6a01577" is_pre_election: true
I20260926 22:49:03.601449 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733898382f1844f6b43db71f1ce7906c" candidate_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" is_pre_election: true
I20260926 22:49:03.601511 15797 raft_consensus.cc:2454] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d44dafd82443baa1c13f91ebd2f01e in term 0.
I20260926 22:49:03.601540 16071 raft_consensus.cc:2454] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d44dafd82443baa1c13f91ebd2f01e in term 0.
I20260926 22:49:03.601673 15868 leader_election.cc:302] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1781b828b532420f9017d00ab6a01577, a0d44dafd82443baa1c13f91ebd2f01e; no voters:
I20260926 22:49:03.601818 16151 raft_consensus.cc:2781] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:03.601866 16151 raft_consensus.cc:482] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:03.601879 16151 raft_consensus.cc:3037] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.602473 16151 raft_consensus.cc:504] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.602577 16151 leader_election.cc:288] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [CANDIDATE]: Term 1 election: Requested vote from peers 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:03.602698 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733898382f1844f6b43db71f1ce7906c" candidate_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1781b828b532420f9017d00ab6a01577"
I20260926 22:49:03.602733 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "733898382f1844f6b43db71f1ce7906c" candidate_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd"
I20260926 22:49:03.602774 15797 raft_consensus.cc:3037] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.602804 16071 raft_consensus.cc:3037] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.603349 15797 raft_consensus.cc:2454] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d44dafd82443baa1c13f91ebd2f01e in term 1.
I20260926 22:49:03.603358 16071 raft_consensus.cc:2454] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d44dafd82443baa1c13f91ebd2f01e in term 1.
I20260926 22:49:03.603524 15870 leader_election.cc:302] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d44dafd82443baa1c13f91ebd2f01e, bf9fb964b20e46ccab6b52cdefc5b6dd; no voters:
I20260926 22:49:03.603616 16151 raft_consensus.cc:2781] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:03.603740 16151 raft_consensus.cc:686] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [term 1 LEADER]: Becoming Leader. State: Replica: a0d44dafd82443baa1c13f91ebd2f01e, State: Running, Role: LEADER
I20260926 22:49:03.603820 16151 consensus_queue.cc:237] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.604216 16166 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: TxnStatusTablet state changed. Reason: New leader a0d44dafd82443baa1c13f91ebd2f01e. Latest consensus state: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.604293 16166 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:03.604447 16168 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 22:49:03.604545 16168 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 22:49:03.604501 15640 catalog_manager.cc:5495] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2). New cstate: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:03.620558 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.617931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.13.148.2:33593)
0926 22:49:03.617973 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.617975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.618073 (+ 98us) server_negotiation.cc:407] Connection header received
0926 22:49:03.618160 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.618161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.618174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.618226 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:03.618334 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.618763 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.619804 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.619977 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.620038 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.620243 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:03.620350 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:03.620406 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.620464 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.620497 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:49:03.620585 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.617874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33593 (local address 127.0.0.1:43258)
0926 22:49:03.618007 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.618034 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.618093 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.618235 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.618236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.618237 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:03.618286 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.618288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.618823 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.618823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.619726 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.619729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.620047 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.620048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.620131 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.620148 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:03.620406 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:03.620412 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.620461 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.620505 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 22:49:03.622411 16071 raft_consensus.cc:1261] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Refusing update from remote peer a0d44dafd82443baa1c13f91ebd2f01e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:03.622439 15797 raft_consensus.cc:1261] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Refusing update from remote peer a0d44dafd82443baa1c13f91ebd2f01e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:03.622699 16151 consensus_queue.cc:1035] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:03.622810 16166 consensus_queue.cc:1035] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:03.623853 16146 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: TxnStatusTablet state changed. Reason: New leader a0d44dafd82443baa1c13f91ebd2f01e. Latest consensus state: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.623934 16146 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:03.623921 16149 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: TxnStatusTablet state changed. Reason: New leader a0d44dafd82443baa1c13f91ebd2f01e. Latest consensus state: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.623999 16149 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:03.624930 16151 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.625005 16151 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:03.625044 16151 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.625079 16151 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P a0d44dafd82443baa1c13f91ebd2f01e: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:03.625139 16149 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.625229 16149 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P bf9fb964b20e46ccab6b52cdefc5b6dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:03.625670 16171 mvcc.cc:205] Tried to move back new op lower bound from 7333736217075290112 to 7333736217001828352. Current Snapshot: MvccSnapshot[applied={T|T < 7333736217075290112}]
I20260926 22:49:03.625680 16146 tablet_replica.cc:439] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } } }
I20260926 22:49:03.625778 16146 tablet_replica.cc:442] T 733898382f1844f6b43db71f1ce7906c P 1781b828b532420f9017d00ab6a01577: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:03.626024 16172 mvcc.cc:205] Tried to move back new op lower bound from 7333736217075290112 to 7333736217001828352. Current Snapshot: MvccSnapshot[applied={T|T < 7333736217075290112}]
I20260926 22:49:03.630239 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.627751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.13.148.1:45121)
0926 22:49:03.627800 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.627802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.627837 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:49:03.627910 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.627911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.627923 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.627974 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:03.628088 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.628532 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.629587 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.629805 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.629850 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.630001 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:03.630081 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:03.630128 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.630178 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.630192 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:49:03.634855 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.632443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:36567 (local address 127.13.148.2:33593)
0926 22:49:03.632482 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.632483 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.632574 (+ 91us) server_negotiation.cc:407] Connection header received
0926 22:49:03.632610 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.632611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.632619 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.632680 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:03.632760 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.633464 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.634177 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.634480 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.634547 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.634664 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:03.634693 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:03.634739 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.634774 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.634789 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:49:03.634862 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.632373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33593 (local address 127.13.148.1:36567)
0926 22:49:03.632518 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.632525 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.632572 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.632684 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.632685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.632686 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:03.632726 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.632727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.633524 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.633524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.634114 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.634116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.634543 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.634544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.634585 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.634595 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:03.634741 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:03.634742 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.634770 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.634808 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260926 22:49:03.636802 16071 raft_consensus.cc:1261] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Refusing update from remote peer 1781b828b532420f9017d00ab6a01577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:03.636883 15937 raft_consensus.cc:1261] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Refusing update from remote peer 1781b828b532420f9017d00ab6a01577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:03.637141 16153 consensus_queue.cc:1035] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:03.637265 16146 consensus_queue.cc:1035] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:03.645855 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.643321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33835 (local address 127.13.148.1:45121)
0926 22:49:03.643353 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:03.643354 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:03.643483 (+ 129us) server_negotiation.cc:407] Connection header received
0926 22:49:03.643532 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:03.643533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:03.643541 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:03.643588 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:03.643701 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.644282 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.645230 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.645434 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.645484 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.645621 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:03.645649 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:03.645692 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:03.645761 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 22:49:03.645778 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 22:49:03.645880 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:03.643293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45121 (local address 127.13.148.2:33835)
0926 22:49:03.643422 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:03.643431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:03.643482 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:03.643592 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:03.643592 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:03.643594 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:03.643650 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.643652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.644356 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.644357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.645167 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:03.645169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:03.645489 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:03.645490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:03.645549 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:03.645563 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:03.645697 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:03.645699 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:49:03.645751 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:49:03.645788 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260926 22:49:03.654582 15638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:03.654682 15638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:03.655332 15638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: Sending DeleteTablet for 3 replicas of tablet bcb4eb4d5c7e469eb0f698d4038c2e70
I20260926 22:49:03.655642 16057 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb4eb4d5c7e469eb0f698d4038c2e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:40438
I20260926 22:49:03.655666 15917 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb4eb4d5c7e469eb0f698d4038c2e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43242
I20260926 22:49:03.656001 15777 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb4eb4d5c7e469eb0f698d4038c2e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:03 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:42808
I20260926 22:49:03.656257 16191 tablet_replica.cc:330] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: stopping tablet replica
I20260926 22:49:03.656179 15638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:03.656337 15638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:03.656498 16193 tablet_replica.cc:330] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: stopping tablet replica
I20260926 22:49:03.656631 16193 raft_consensus.cc:2229] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:03.656764 16191 raft_consensus.cc:2229] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:03.656911 16193 raft_consensus.cc:2258] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:03.657007 16191 raft_consensus.cc:2258] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:03.657331 16057 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7b1be6adf14386a3be539a322010e6 (DEFAULT_TABLE table=test-table [id=b690dfa12d01421994a3151a4c2d8053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:03.657454 16057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7b1be6adf14386a3be539a322010e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.657642 16193 ts_tablet_manager.cc:1905] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:03.658885 16145 tablet_bootstrap.cc:492] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: Bootstrap starting.
I20260926 22:49:03.659054 16192 tablet_replica.cc:330] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: stopping tablet replica
I20260926 22:49:03.659158 16192 raft_consensus.cc:2229] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:03.659215 16192 raft_consensus.cc:2258] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:03.659276 16145 tablet_bootstrap.cc:654] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.659703 16193 ts_tablet_manager.cc:1918] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:03.659898 15917 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7b1be6adf14386a3be539a322010e6 (DEFAULT_TABLE table=test-table [id=b690dfa12d01421994a3151a4c2d8053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:03.659999 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7b1be6adf14386a3be539a322010e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.661255 16148 tablet_bootstrap.cc:492] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: Bootstrap starting.
I20260926 22:49:03.661626 16148 tablet_bootstrap.cc:654] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.659708 15777 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7b1be6adf14386a3be539a322010e6 (DEFAULT_TABLE table=test-table [id=b690dfa12d01421994a3151a4c2d8053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:03.661849 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7b1be6adf14386a3be539a322010e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:03.662420 16192 ts_tablet_manager.cc:1905] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:03.662967 16143 tablet_bootstrap.cc:492] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: Bootstrap starting.
I20260926 22:49:03.663357 16143 tablet_bootstrap.cc:654] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:03.664009 16193 log.cc:1196] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/bcb4eb4d5c7e469eb0f698d4038c2e70
I20260926 22:49:03.664139 16192 ts_tablet_manager.cc:1918] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:03.664208 16192 log.cc:1196] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/bcb4eb4d5c7e469eb0f698d4038c2e70
I20260926 22:49:03.664443 16193 ts_tablet_manager.cc:1939] T bcb4eb4d5c7e469eb0f698d4038c2e70 P 1781b828b532420f9017d00ab6a01577: Deleting consensus metadata
I20260926 22:49:03.664474 16192 ts_tablet_manager.cc:1939] T bcb4eb4d5c7e469eb0f698d4038c2e70 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting consensus metadata
I20260926 22:49:03.664837 15627 catalog_manager.cc:4841] TS a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593): tablet bcb4eb4d5c7e469eb0f698d4038c2e70 (table test-table [id=358dcc95d56044aebe012c32bee37357]) successfully deleted
I20260926 22:49:03.664862 15624 catalog_manager.cc:4841] TS 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121): tablet bcb4eb4d5c7e469eb0f698d4038c2e70 (table test-table [id=358dcc95d56044aebe012c32bee37357]) successfully deleted
I20260926 22:49:03.665118 16191 ts_tablet_manager.cc:1905] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:03.665377 16143 tablet_bootstrap.cc:492] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: No bootstrap required, opened a new log
I20260926 22:49:03.665419 16143 ts_tablet_manager.cc:1397] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:49:03.665548 16143 raft_consensus.cc:348] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.665606 16143 raft_consensus.cc:374] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.665624 16143 raft_consensus.cc:729] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1781b828b532420f9017d00ab6a01577, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.665663 16143 consensus_queue.cc:260] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.665772 16143 ts_tablet_manager.cc:1428] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.666582 16191 ts_tablet_manager.cc:1918] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:03.666651 16191 log.cc:1196] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/bcb4eb4d5c7e469eb0f698d4038c2e70
I20260926 22:49:03.666873 16191 ts_tablet_manager.cc:1939] T bcb4eb4d5c7e469eb0f698d4038c2e70 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting consensus metadata
I20260926 22:49:03.667021 16148 tablet_bootstrap.cc:492] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: No bootstrap required, opened a new log
I20260926 22:49:03.667059 16148 ts_tablet_manager.cc:1397] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 22:49:03.667172 16148 raft_consensus.cc:348] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.667238 16148 raft_consensus.cc:374] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.667225 15626 catalog_manager.cc:4841] TS bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681): tablet bcb4eb4d5c7e469eb0f698d4038c2e70 (table test-table [id=358dcc95d56044aebe012c32bee37357]) successfully deleted
I20260926 22:49:03.667258 16148 raft_consensus.cc:729] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d44dafd82443baa1c13f91ebd2f01e, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.667295 16148 consensus_queue.cc:260] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.667369 16148 ts_tablet_manager.cc:1428] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:03.667873 16145 tablet_bootstrap.cc:492] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: No bootstrap required, opened a new log
I20260926 22:49:03.667920 16145 ts_tablet_manager.cc:1397] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260926 22:49:03.668051 16145 raft_consensus.cc:348] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.668115 16145 raft_consensus.cc:374] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:03.668141 16145 raft_consensus.cc:729] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fb964b20e46ccab6b52cdefc5b6dd, State: Initialized, Role: FOLLOWER
I20260926 22:49:03.668179 16145 consensus_queue.cc:260] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.668269 16145 ts_tablet_manager.cc:1428] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:03.828631 16153 raft_consensus.cc:482] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:03.828735 16153 raft_consensus.cc:504] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.828902 16153 leader_election.cc:288] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:03.829129 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7b1be6adf14386a3be539a322010e6" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" is_pre_election: true
I20260926 22:49:03.829128 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7b1be6adf14386a3be539a322010e6" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" is_pre_election: true
I20260926 22:49:03.829216 15937 raft_consensus.cc:2454] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 0.
I20260926 22:49:03.829239 16071 raft_consensus.cc:2454] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 0.
I20260926 22:49:03.829414 15731 leader_election.cc:302] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1781b828b532420f9017d00ab6a01577, a0d44dafd82443baa1c13f91ebd2f01e; no voters:
I20260926 22:49:03.829535 16153 raft_consensus.cc:2781] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:03.829583 16153 raft_consensus.cc:482] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:03.829603 16153 raft_consensus.cc:3037] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.830318 16153 raft_consensus.cc:504] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.830432 16153 leader_election.cc:288] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 election: Requested vote from peers a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:03.830597 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7b1be6adf14386a3be539a322010e6" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd"
I20260926 22:49:03.830583 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7b1be6adf14386a3be539a322010e6" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d44dafd82443baa1c13f91ebd2f01e"
I20260926 22:49:03.830662 16071 raft_consensus.cc:3037] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.830664 15937 raft_consensus.cc:3037] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:03.831243 16071 raft_consensus.cc:2454] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 1.
I20260926 22:49:03.831267 15937 raft_consensus.cc:2454] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 1.
I20260926 22:49:03.831414 15730 leader_election.cc:302] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1781b828b532420f9017d00ab6a01577, bf9fb964b20e46ccab6b52cdefc5b6dd; no voters:
I20260926 22:49:03.831511 16153 raft_consensus.cc:2781] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:03.831584 16153 raft_consensus.cc:686] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 1 LEADER]: Becoming Leader. State: Replica: 1781b828b532420f9017d00ab6a01577, State: Running, Role: LEADER
I20260926 22:49:03.831634 16153 consensus_queue.cc:237] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:03.832185 15638 catalog_manager.cc:5495] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1781b828b532420f9017d00ab6a01577 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "1781b828b532420f9017d00ab6a01577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:03.845120 15937 raft_consensus.cc:1261] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Refusing update from remote peer 1781b828b532420f9017d00ab6a01577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:03.845120 16071 raft_consensus.cc:1261] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Refusing update from remote peer 1781b828b532420f9017d00ab6a01577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:03.845330 16186 consensus_queue.cc:1035] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:03.845443 16153 consensus_queue.cc:1035] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:03.846777 16173 mvcc.cc:205] Tried to move back new op lower bound from 7333736217988263936 to 7333736217934626816. Current Snapshot: MvccSnapshot[applied={T|T < 7333736217988263936}]
W20260926 22:49:03.849681 16173 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 22:49:05.366240 15639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:05.366335 15639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:05.367041 15639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: Sending DeleteTablet for 3 replicas of tablet 2d7b1be6adf14386a3be539a322010e6
I20260926 22:49:05.367352 15777 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7b1be6adf14386a3be539a322010e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:05 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:42808
I20260926 22:49:05.367524 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7b1be6adf14386a3be539a322010e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:05 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43242
I20260926 22:49:05.367695 16057 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7b1be6adf14386a3be539a322010e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:05 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:40438
I20260926 22:49:05.367707 16222 tablet_replica.cc:330] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: stopping tablet replica
I20260926 22:49:05.367769 16222 raft_consensus.cc:2229] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:05.367816 16222 raft_consensus.cc:2258] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:05.369210 16222 ts_tablet_manager.cc:1905] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:05.369801 16221 tablet_replica.cc:330] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: stopping tablet replica
I20260926 22:49:05.369858 16221 raft_consensus.cc:2229] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:05.369918 16221 raft_consensus.cc:2258] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:05.370116 16221 ts_tablet_manager.cc:1905] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:05.371117 16222 ts_tablet_manager.cc:1918] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:49:05.371168 16222 log.cc:1196] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/2d7b1be6adf14386a3be539a322010e6
I20260926 22:49:05.371356 16222 ts_tablet_manager.cc:1939] T 2d7b1be6adf14386a3be539a322010e6 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting consensus metadata
I20260926 22:49:05.371603 16223 tablet_replica.cc:330] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: stopping tablet replica
I20260926 22:49:05.371670 16223 raft_consensus.cc:2229] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:05.371667 15627 catalog_manager.cc:4841] TS a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593): tablet 2d7b1be6adf14386a3be539a322010e6 (table test-table [id=b690dfa12d01421994a3151a4c2d8053]) successfully deleted
I20260926 22:49:05.371706 16223 raft_consensus.cc:2258] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:05.371918 16221 ts_tablet_manager.cc:1918] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:49:05.371974 16221 log.cc:1196] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/2d7b1be6adf14386a3be539a322010e6
I20260926 22:49:05.372182 16221 ts_tablet_manager.cc:1939] T 2d7b1be6adf14386a3be539a322010e6 P 1781b828b532420f9017d00ab6a01577: Deleting consensus metadata
I20260926 22:49:05.372504 15624 catalog_manager.cc:4841] TS 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121): tablet 2d7b1be6adf14386a3be539a322010e6 (table test-table [id=b690dfa12d01421994a3151a4c2d8053]) successfully deleted
I20260926 22:49:05.372874 16223 ts_tablet_manager.cc:1905] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:05.374125 16223 ts_tablet_manager.cc:1918] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:49:05.374186 16223 log.cc:1196] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/2d7b1be6adf14386a3be539a322010e6
I20260926 22:49:05.374385 16223 ts_tablet_manager.cc:1939] T 2d7b1be6adf14386a3be539a322010e6 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting consensus metadata
I20260926 22:49:05.374684 15626 catalog_manager.cc:4841] TS bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681): tablet 2d7b1be6adf14386a3be539a322010e6 (table test-table [id=b690dfa12d01421994a3151a4c2d8053]) successfully deleted
Sep 26 22:49:05 dist-test-slave-98hb krb5kdc[15595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:49:05 dist-test-slave-98hb krb5kdc[15595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:05.390012 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:05.383342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.13.148.62:46715)
0926 22:49:05.383482 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:05.383484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:05.383493 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:05.383551 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:05.383552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:05.383562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:05.383619 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:05.384139 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.384662 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.385380 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.385576 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.385627 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:05.387688 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:05.387690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:05.387695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:05.387702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:05.388982 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:05.389279 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:05.389280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:05.389280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:05.389343 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:05.389498 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:05.389499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:05.389499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:05.389600 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:05.389687 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:05.389838 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 22:49:05.389917 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260926 22:49:05.391227 15639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:05.391378 15639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:05.392577 15777 tablet_service.cc:1467] Processing CreateTablet for tablet 7de14151901c4851b2a70610bbbdfbd1 (DEFAULT_TABLE table=test-table [id=412717cab62e419fac095520b54840ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:05.392601 15916 tablet_service.cc:1467] Processing CreateTablet for tablet 7de14151901c4851b2a70610bbbdfbd1 (DEFAULT_TABLE table=test-table [id=412717cab62e419fac095520b54840ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:05.392668 16057 tablet_service.cc:1467] Processing CreateTablet for tablet 7de14151901c4851b2a70610bbbdfbd1 (DEFAULT_TABLE table=test-table [id=412717cab62e419fac095520b54840ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:05.392730 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de14151901c4851b2a70610bbbdfbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:05.392772 16057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de14151901c4851b2a70610bbbdfbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:05.392791 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7de14151901c4851b2a70610bbbdfbd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:05.393766 16232 tablet_bootstrap.cc:492] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: Bootstrap starting.
I20260926 22:49:05.394090 16233 tablet_bootstrap.cc:492] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: Bootstrap starting.
I20260926 22:49:05.394124 16232 tablet_bootstrap.cc:654] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:05.394543 16233 tablet_bootstrap.cc:654] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:05.394690 16232 tablet_bootstrap.cc:492] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: No bootstrap required, opened a new log
I20260926 22:49:05.394738 16232 ts_tablet_manager.cc:1397] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:05.394850 16234 tablet_bootstrap.cc:492] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: Bootstrap starting.
I20260926 22:49:05.394883 16232 raft_consensus.cc:348] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.394959 16232 raft_consensus.cc:374] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:05.394989 16232 raft_consensus.cc:729] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1781b828b532420f9017d00ab6a01577, State: Initialized, Role: FOLLOWER
I20260926 22:49:05.395053 16232 consensus_queue.cc:260] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.395159 16234 tablet_bootstrap.cc:654] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:05.395157 16232 ts_tablet_manager.cc:1428] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:05.395263 16233 tablet_bootstrap.cc:492] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: No bootstrap required, opened a new log
I20260926 22:49:05.395311 16233 ts_tablet_manager.cc:1397] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:05.395418 16233 raft_consensus.cc:348] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.395475 16233 raft_consensus.cc:374] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:05.395495 16233 raft_consensus.cc:729] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fb964b20e46ccab6b52cdefc5b6dd, State: Initialized, Role: FOLLOWER
I20260926 22:49:05.395553 16233 consensus_queue.cc:260] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.395706 16233 ts_tablet_manager.cc:1428] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:05.395859 16234 tablet_bootstrap.cc:492] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: No bootstrap required, opened a new log
I20260926 22:49:05.395913 16234 ts_tablet_manager.cc:1397] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:05.396035 16234 raft_consensus.cc:348] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.396122 16234 raft_consensus.cc:374] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:05.396142 16234 raft_consensus.cc:729] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d44dafd82443baa1c13f91ebd2f01e, State: Initialized, Role: FOLLOWER
I20260926 22:49:05.396184 16234 consensus_queue.cc:260] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.396317 16234 ts_tablet_manager.cc:1428] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:05.523787 16202 raft_consensus.cc:482] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:05.523880 16202 raft_consensus.cc:504] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.524037 16202 leader_election.cc:288] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:05.524259 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de14151901c4851b2a70610bbbdfbd1" candidate_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1781b828b532420f9017d00ab6a01577" is_pre_election: true
I20260926 22:49:05.524253 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de14151901c4851b2a70610bbbdfbd1" candidate_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" is_pre_election: true
I20260926 22:49:05.524356 15797 raft_consensus.cc:2454] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d44dafd82443baa1c13f91ebd2f01e in term 0.
I20260926 22:49:05.524356 16071 raft_consensus.cc:2454] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0d44dafd82443baa1c13f91ebd2f01e in term 0.
I20260926 22:49:05.524502 15870 leader_election.cc:302] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d44dafd82443baa1c13f91ebd2f01e, bf9fb964b20e46ccab6b52cdefc5b6dd; no voters:
I20260926 22:49:05.524595 16202 raft_consensus.cc:2781] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:05.524633 16202 raft_consensus.cc:482] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:05.524648 16202 raft_consensus.cc:3037] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:05.525367 16202 raft_consensus.cc:504] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.525489 16202 leader_election.cc:288] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [CANDIDATE]: Term 1 election: Requested vote from peers 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:05.525619 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de14151901c4851b2a70610bbbdfbd1" candidate_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd"
I20260926 22:49:05.525609 15797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de14151901c4851b2a70610bbbdfbd1" candidate_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1781b828b532420f9017d00ab6a01577"
I20260926 22:49:05.525698 16071 raft_consensus.cc:3037] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:05.525698 15797 raft_consensus.cc:3037] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:05.526438 15797 raft_consensus.cc:2454] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d44dafd82443baa1c13f91ebd2f01e in term 1.
I20260926 22:49:05.526438 16071 raft_consensus.cc:2454] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0d44dafd82443baa1c13f91ebd2f01e in term 1.
I20260926 22:49:05.526612 15868 leader_election.cc:302] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1781b828b532420f9017d00ab6a01577, a0d44dafd82443baa1c13f91ebd2f01e; no voters:
I20260926 22:49:05.526710 16202 raft_consensus.cc:2781] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:05.526762 16202 raft_consensus.cc:686] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 LEADER]: Becoming Leader. State: Replica: a0d44dafd82443baa1c13f91ebd2f01e, State: Running, Role: LEADER
I20260926 22:49:05.526808 16202 consensus_queue.cc:237] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.527393 15638 catalog_manager.cc:5495] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e reported cstate change: term changed from 0 to 1, leader changed from <none> to a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2). New cstate: current_term: 1 leader_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:05.546440 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:05.544406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.13.148.2:33593)
0926 22:49:05.544534 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:05.544535 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:05.544541 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:49:05.544556 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:05.544556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:05.544564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:05.544610 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:05.544759 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.545223 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.545957 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.546136 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.546180 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:05.546273 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:05.546297 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:05.546327 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:05.546356 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 22:49:05.546370 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":84,"threads_started":1}
I20260926 22:49:05.550205 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:05.548364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:34291 (local address 127.13.148.2:33593)
0926 22:49:05.548391 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:05.548392 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:05.548441 (+ 49us) server_negotiation.cc:407] Connection header received
0926 22:49:05.548493 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:05.548494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:05.548502 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:05.548531 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:05.548609 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.549043 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.549702 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.549913 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.549954 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:05.550050 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:05.550071 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:05.550101 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:05.550130 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 22:49:05.550144 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 22:49:05.550251 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:05.548291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33593 (local address 127.13.148.2:34291)
0926 22:49:05.548392 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:05.548401 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:05.548453 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:05.548541 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:05.548541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:05.548542 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:05.548578 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:05.548579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.549103 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.549104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:05.549646 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:05.549647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.549966 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.549967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:05.550004 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:05.550007 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:05.550101 (+ 94us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:05.550102 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:49:05.550129 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 22:49:05.550159 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":59,"threads_started":1}
I20260926 22:49:05.551764 16071 raft_consensus.cc:1261] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Refusing update from remote peer a0d44dafd82443baa1c13f91ebd2f01e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:05.551788 15797 raft_consensus.cc:1261] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Refusing update from remote peer a0d44dafd82443baa1c13f91ebd2f01e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:05.551944 16202 consensus_queue.cc:1035] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:05.552049 16207 consensus_queue.cc:1035] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:05.561630 15638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:05.561718 15638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:05.562322 15638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: Sending DeleteTablet for 3 replicas of tablet 7de14151901c4851b2a70610bbbdfbd1
I20260926 22:49:05.562541 15777 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de14151901c4851b2a70610bbbdfbd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:05 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:42808
I20260926 22:49:05.562569 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de14151901c4851b2a70610bbbdfbd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:05 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43242
I20260926 22:49:05.562637 16222 tablet_replica.cc:330] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: stopping tablet replica
I20260926 22:49:05.562640 16057 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de14151901c4851b2a70610bbbdfbd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:05 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:40438
I20260926 22:49:05.562680 16222 raft_consensus.cc:2229] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:05.562713 16221 tablet_replica.cc:330] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: stopping tablet replica
I20260926 22:49:05.562753 16222 raft_consensus.cc:2258] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:05.562754 16221 raft_consensus.cc:2229] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:05.562783 16221 raft_consensus.cc:2258] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:05.562752 15638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 22:49:05.562837 16223 tablet_replica.cc:330] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: stopping tablet replica
I20260926 22:49:05.562877 16223 raft_consensus.cc:2229] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 22:49:05.562878 15638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:05.562913 16223 raft_consensus.cc:2258] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:05.563058 16222 ts_tablet_manager.cc:1905] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:05.563247 16223 ts_tablet_manager.cc:1905] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:05.564682 16223 ts_tablet_manager.cc:1918] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:05.564682 16222 ts_tablet_manager.cc:1918] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:05.564738 16222 log.cc:1196] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/7de14151901c4851b2a70610bbbdfbd1
I20260926 22:49:05.564738 16223 log.cc:1196] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/7de14151901c4851b2a70610bbbdfbd1
I20260926 22:49:05.564937 16222 ts_tablet_manager.cc:1939] T 7de14151901c4851b2a70610bbbdfbd1 P a0d44dafd82443baa1c13f91ebd2f01e: Deleting consensus metadata
I20260926 22:49:05.564937 16223 ts_tablet_manager.cc:1939] T 7de14151901c4851b2a70610bbbdfbd1 P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting consensus metadata
I20260926 22:49:05.565111 16221 ts_tablet_manager.cc:1905] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:05.565228 15626 catalog_manager.cc:4841] TS bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681): tablet 7de14151901c4851b2a70610bbbdfbd1 (table test-table [id=412717cab62e419fac095520b54840ee]) successfully deleted
I20260926 22:49:05.565291 16057 tablet_service.cc:1467] Processing CreateTablet for tablet 198931c32cc14e238d9c756454fabf3f (DEFAULT_TABLE table=test-table [id=7532f034873a4500a7f5af177810cd1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:05.565385 16057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198931c32cc14e238d9c756454fabf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:05.565662 15627 catalog_manager.cc:4841] TS a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593): tablet 7de14151901c4851b2a70610bbbdfbd1 (table test-table [id=412717cab62e419fac095520b54840ee]) successfully deleted
I20260926 22:49:05.565781 15916 tablet_service.cc:1467] Processing CreateTablet for tablet 198931c32cc14e238d9c756454fabf3f (DEFAULT_TABLE table=test-table [id=7532f034873a4500a7f5af177810cd1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:05.565873 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198931c32cc14e238d9c756454fabf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:05.566524 16221 ts_tablet_manager.cc:1918] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:05.566567 16221 log.cc:1196] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/7de14151901c4851b2a70610bbbdfbd1
I20260926 22:49:05.566740 16221 ts_tablet_manager.cc:1939] T 7de14151901c4851b2a70610bbbdfbd1 P 1781b828b532420f9017d00ab6a01577: Deleting consensus metadata
I20260926 22:49:05.567003 16234 tablet_bootstrap.cc:492] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: Bootstrap starting.
I20260926 22:49:05.567021 16233 tablet_bootstrap.cc:492] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: Bootstrap starting.
I20260926 22:49:05.567317 16234 tablet_bootstrap.cc:654] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:05.567489 16233 tablet_bootstrap.cc:654] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:05.567900 16234 tablet_bootstrap.cc:492] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: No bootstrap required, opened a new log
I20260926 22:49:05.567948 16234 ts_tablet_manager.cc:1397] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:05.568050 15777 tablet_service.cc:1467] Processing CreateTablet for tablet 198931c32cc14e238d9c756454fabf3f (DEFAULT_TABLE table=test-table [id=7532f034873a4500a7f5af177810cd1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:05.568042 16234 raft_consensus.cc:348] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.568100 16234 raft_consensus.cc:374] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:05.568120 16234 raft_consensus.cc:729] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d44dafd82443baa1c13f91ebd2f01e, State: Initialized, Role: FOLLOWER
I20260926 22:49:05.568147 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198931c32cc14e238d9c756454fabf3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:05.568161 16234 consensus_queue.cc:260] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.568248 16234 ts_tablet_manager.cc:1428] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:05.568931 16232 tablet_bootstrap.cc:492] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577: Bootstrap starting.
I20260926 22:49:05.569010 15624 catalog_manager.cc:4841] TS 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121): tablet 7de14151901c4851b2a70610bbbdfbd1 (table test-table [id=412717cab62e419fac095520b54840ee]) successfully deleted
I20260926 22:49:05.569211 16232 tablet_bootstrap.cc:654] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:05.569692 16233 tablet_bootstrap.cc:492] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: No bootstrap required, opened a new log
I20260926 22:49:05.569774 16233 ts_tablet_manager.cc:1397] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:49:05.569907 16233 raft_consensus.cc:348] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.569989 16233 raft_consensus.cc:374] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:05.570013 16233 raft_consensus.cc:729] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fb964b20e46ccab6b52cdefc5b6dd, State: Initialized, Role: FOLLOWER
I20260926 22:49:05.570050 16233 consensus_queue.cc:260] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.570128 16233 ts_tablet_manager.cc:1428] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:05.570319 16232 tablet_bootstrap.cc:492] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577: No bootstrap required, opened a new log
I20260926 22:49:05.570369 16232 ts_tablet_manager.cc:1397] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 22:49:05.570470 16232 raft_consensus.cc:348] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.570549 16232 raft_consensus.cc:374] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:05.570577 16232 raft_consensus.cc:729] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1781b828b532420f9017d00ab6a01577, State: Initialized, Role: FOLLOWER
I20260926 22:49:05.570621 16232 consensus_queue.cc:260] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.570703 16232 ts_tablet_manager.cc:1428] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:05.651051 16212 raft_consensus.cc:482] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:05.651142 16212 raft_consensus.cc:504] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.651319 16212 leader_election.cc:288] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:05.651482 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198931c32cc14e238d9c756454fabf3f" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" is_pre_election: true
I20260926 22:49:05.651573 15937 raft_consensus.cc:2454] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 0.
I20260926 22:49:05.651564 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198931c32cc14e238d9c756454fabf3f" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" is_pre_election: true
I20260926 22:49:05.651647 16071 raft_consensus.cc:2454] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 0.
I20260926 22:49:05.651728 15731 leader_election.cc:302] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1781b828b532420f9017d00ab6a01577, a0d44dafd82443baa1c13f91ebd2f01e; no voters:
I20260926 22:49:05.651829 16212 raft_consensus.cc:2781] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:05.651868 16212 raft_consensus.cc:482] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:05.651886 16212 raft_consensus.cc:3037] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:05.652544 16212 raft_consensus.cc:504] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.652666 16212 leader_election.cc:288] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 election: Requested vote from peers a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593), bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681)
I20260926 22:49:05.652814 15937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198931c32cc14e238d9c756454fabf3f" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d44dafd82443baa1c13f91ebd2f01e"
I20260926 22:49:05.652853 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "198931c32cc14e238d9c756454fabf3f" candidate_uuid: "1781b828b532420f9017d00ab6a01577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd"
I20260926 22:49:05.652902 15937 raft_consensus.cc:3037] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:05.652915 16071 raft_consensus.cc:3037] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:05.653595 15937 raft_consensus.cc:2454] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 1.
I20260926 22:49:05.653595 16071 raft_consensus.cc:2454] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1781b828b532420f9017d00ab6a01577 in term 1.
I20260926 22:49:05.653776 15731 leader_election.cc:302] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1781b828b532420f9017d00ab6a01577, a0d44dafd82443baa1c13f91ebd2f01e; no voters:
I20260926 22:49:05.653873 16212 raft_consensus.cc:2781] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:05.653914 16212 raft_consensus.cc:686] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 1 LEADER]: Becoming Leader. State: Replica: 1781b828b532420f9017d00ab6a01577, State: Running, Role: LEADER
I20260926 22:49:05.653962 16212 consensus_queue.cc:237] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } }
I20260926 22:49:05.654517 15638 catalog_manager.cc:5495] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1781b828b532420f9017d00ab6a01577 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "1781b828b532420f9017d00ab6a01577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1781b828b532420f9017d00ab6a01577" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:05.663087 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:05.660731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.13.148.1:45121)
0926 22:49:05.660876 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:05.660877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:05.660883 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:49:05.660901 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:05.660901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:05.660909 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:05.660960 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:05.661090 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.661555 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.662548 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:05.662733 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:05.662780 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:05.662913 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:05.662938 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:05.662967 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:05.663018 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 22:49:05.663032 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 22:49:05.664914 16071 raft_consensus.cc:1261] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Refusing update from remote peer 1781b828b532420f9017d00ab6a01577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:05.665087 15937 raft_consensus.cc:1261] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Refusing update from remote peer 1781b828b532420f9017d00ab6a01577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:05.665086 16215 consensus_queue.cc:1035] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fb964b20e46ccab6b52cdefc5b6dd" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:05.665259 16215 consensus_queue.cc:1035] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d44dafd82443baa1c13f91ebd2f01e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 22:49:05.668658 16204 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 22:49:07.184533 15639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:07.184634 15639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:07.185376 15639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 181d8e2ecedc44608d282a19351efae1: Sending DeleteTablet for 3 replicas of tablet 198931c32cc14e238d9c756454fabf3f
I20260926 22:49:07.185587 15777 tablet_service.cc:1514] Processing DeleteTablet for tablet 198931c32cc14e238d9c756454fabf3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:07 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:42808
I20260926 22:49:07.185797 16279 tablet_replica.cc:330] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577: stopping tablet replica
I20260926 22:49:07.185789 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 15712
I20260926 22:49:07.185858 16279 raft_consensus.cc:2229] T 198931c32cc14e238d9c756454fabf3f P 1781b828b532420f9017d00ab6a01577 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:07.186215 16057 tablet_service.cc:1514] Processing DeleteTablet for tablet 198931c32cc14e238d9c756454fabf3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:07 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:40438
I20260926 22:49:07.186376 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet 198931c32cc14e238d9c756454fabf3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:07 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43242
I20260926 22:49:07.186393 16280 tablet_replica.cc:330] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: stopping tablet replica
I20260926 22:49:07.186435 16280 raft_consensus.cc:2229] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:07.186460 16280 raft_consensus.cc:2258] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:07.186543 16281 tablet_replica.cc:330] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: stopping tablet replica
I20260926 22:49:07.186590 16281 raft_consensus.cc:2229] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:07.186652 16281 raft_consensus.cc:2258] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:07.186731 16280 ts_tablet_manager.cc:1905] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:07.186848 16281 ts_tablet_manager.cc:1905] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:07.188431 16280 ts_tablet_manager.cc:1918] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:49:07.188493 16280 log.cc:1196] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/198931c32cc14e238d9c756454fabf3f
I20260926 22:49:07.188694 16280 ts_tablet_manager.cc:1939] T 198931c32cc14e238d9c756454fabf3f P bf9fb964b20e46ccab6b52cdefc5b6dd: Deleting consensus metadata
I20260926 22:49:07.188663 16281 ts_tablet_manager.cc:1918] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 22:49:07.188827 16281 log.cc:1196] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/198931c32cc14e238d9c756454fabf3f
I20260926 22:49:07.189081 16281 ts_tablet_manager.cc:1939] T 198931c32cc14e238d9c756454fabf3f P a0d44dafd82443baa1c13f91ebd2f01e: Deleting consensus metadata
I20260926 22:49:07.189258 15626 catalog_manager.cc:4841] TS bf9fb964b20e46ccab6b52cdefc5b6dd (127.13.148.3:36681): tablet 198931c32cc14e238d9c756454fabf3f (table test-table [id=7532f034873a4500a7f5af177810cd1a]) successfully deleted
I20260926 22:49:07.189363 15627 catalog_manager.cc:4841] TS a0d44dafd82443baa1c13f91ebd2f01e (127.13.148.2:33593): tablet 198931c32cc14e238d9c756454fabf3f (table test-table [id=7532f034873a4500a7f5af177810cd1a]) successfully deleted
W20260926 22:49:07.193261 15624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:45121 (error 108)
I20260926 22:49:07.193416 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 15852
I20260926 22:49:07.195250 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:07.194971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45121 (local address 127.0.0.1:42828)
0926 22:49:07.195099 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:07.195179 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
W20260926 22:49:07.195559 15624 catalog_manager.cc:4584] TS 1781b828b532420f9017d00ab6a01577 (127.13.148.1:45121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 198931c32cc14e238d9c756454fabf3f: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45121: connect: Connection refused (error 111)
I20260926 22:49:07.199673 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 15992
I20260926 22:49:07.204295 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 15608
2026-09-26T22:49:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5669 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:07 dist-test-slave-98hb krb5kdc[16286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:07 dist-test-slave-98hb krb5kdc[16286](info): set up 2 sockets
Sep 26 22:49:07 dist-test-slave-98hb krb5kdc[16286](info): commencing operation
krb5kdc: starting...
W20260926 22:49:08.141567 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:08.247509 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34797
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:34797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:08.324116 16299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:08.324322 16299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:08.324352 16299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:08.325439 16299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:08.325477 16299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:08.325496 16299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:08.325511 16299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:08.325526 16299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:08.327025 16299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:34797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:08.327266 16299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:08.327456 16299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:08.331919 16305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:08.332007 16299 server_base.cc:1034] running on GCE node
W20260926 22:49:08.331936 16304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:08.331969 16307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:08.332369 16299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:08.332628 16299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:08.333784 16299 hybrid_clock.cc:648] HybridClock initialized: now 1790462948333766 us; error 29 us; skew 500 ppm
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:08.339581 16299 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:08.339985 16299 webserver.cc:460] Webserver started at http://127.13.148.62:33071/ using document root <none> and password file <none>
I20260926 22:49:08.340180 16299 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:08.340226 16299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:08.340345 16299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:08.341334 16299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "970add746b0a47319124d4183653bff6"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "6eaa65b0dab6f16fbb52c8092297114a"
server_key_iv: "dc6781730c46a368ab52d2204720a3fc"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.341641 16299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "970add746b0a47319124d4183653bff6"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "6eaa65b0dab6f16fbb52c8092297114a"
server_key_iv: "dc6781730c46a368ab52d2204720a3fc"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.342821 16299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:08.343441 16314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.343581 16299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:08.343658 16299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "970add746b0a47319124d4183653bff6"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "6eaa65b0dab6f16fbb52c8092297114a"
server_key_iv: "dc6781730c46a368ab52d2204720a3fc"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.343717 16299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:08.361853 16299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:08.363487 16299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:08.363616 16299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:08.367148 16299 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:34797
I20260926 22:49:08.367188 16366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:34797 every 8 connection(s)
I20260926 22:49:08.367493 16299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:08.368251 16367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:08.370481 16367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6: Bootstrap starting.
I20260926 22:49:08.370971 16367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:08.371217 16367 log.cc:824] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:08.371291 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 16299
I20260926 22:49:08.371377 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:08.371544 13904 external_mini_cluster.cc:1428] Setting key 44804f9af09cdb459178e22308bd3b60
I20260926 22:49:08.371948 16367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6: No bootstrap required, opened a new log
I20260926 22:49:08.373984 16367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970add746b0a47319124d4183653bff6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34797 } }
I20260926 22:49:08.374120 16367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:08.374151 16367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970add746b0a47319124d4183653bff6, State: Initialized, Role: FOLLOWER
I20260926 22:49:08.374280 16367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970add746b0a47319124d4183653bff6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34797 } }
I20260926 22:49:08.374354 16367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:08.374389 16367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:08.374434 16367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:08.374922 16367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970add746b0a47319124d4183653bff6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34797 } }
I20260926 22:49:08.375011 16367 leader_election.cc:302] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 970add746b0a47319124d4183653bff6; no voters:
I20260926 22:49:08.375154 16367 leader_election.cc:288] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:08.375193 16372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:08.375300 16372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [term 1 LEADER]: Becoming Leader. State: Replica: 970add746b0a47319124d4183653bff6, State: Running, Role: LEADER
I20260926 22:49:08.375393 16367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:08.375434 16372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970add746b0a47319124d4183653bff6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34797 } }
I20260926 22:49:08.375730 16374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "970add746b0a47319124d4183653bff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970add746b0a47319124d4183653bff6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34797 } } }
I20260926 22:49:08.375804 16374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:08.375927 16374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 970add746b0a47319124d4183653bff6. Latest consensus state: current_term: 1 leader_uuid: "970add746b0a47319124d4183653bff6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970add746b0a47319124d4183653bff6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34797 } } }
I20260926 22:49:08.375980 16374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:08.376672 16379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:08.377271 16379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:08.379076 16379 catalog_manager.cc:1347] Generated new cluster ID: a78ed963a7814e29b3c0827f0adc4b87
I20260926 22:49:08.379110 16379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:08.383291 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:08.371975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.13.148.62:34797)
0926 22:49:08.372109 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:08.372117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:08.372144 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:49:08.373037 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:08.373043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:08.373097 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:08.373220 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:08.373972 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.375205 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.376257 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.376898 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.376986 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:08.379479 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:08.379483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:08.379483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:08.379494 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:08.381384 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:08.381803 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:08.381806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:08.381806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:08.381905 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:08.382103 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:08.382103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:08.382104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:08.382389 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:08.382541 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:08.382966 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 22:49:08.383094 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260926 22:49:08.393226 16379 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:08.393929 16379 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:08.398144 16379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 970add746b0a47319124d4183653bff6: Generated new TSK 0
I20260926 22:49:08.398300 16379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:08.426620 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34797
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:08.507146 16393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:08.507346 16393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:08.507368 16393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:08.508507 16393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:08.508546 16393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:08.508572 16393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:08.509981 16393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:08.510205 16393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:08.510403 16393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:08.515348 16399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:08.515400 16398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:08.515553 16393 server_base.cc:1034] running on GCE node
W20260926 22:49:08.515607 16401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:08.515842 16393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:08.516067 16393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:08.517228 16393 hybrid_clock.cc:648] HybridClock initialized: now 1790462948517213 us; error 31 us; skew 500 ppm
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:08.519645 16393 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:08.520064 16393 webserver.cc:460] Webserver started at http://127.13.148.1:41919/ using document root <none> and password file <none>
I20260926 22:49:08.520263 16393 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:08.520310 16393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:08.520431 16393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:08.521453 16393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "6dcf59e9775648a2b50b487741ee7dfa"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "05d9bca5a76bb14782fc5d2a581be53f"
server_key_iv: "af6da17732acf07cbc4b5a17bd38d719"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.521842 16393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "6dcf59e9775648a2b50b487741ee7dfa"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "05d9bca5a76bb14782fc5d2a581be53f"
server_key_iv: "af6da17732acf07cbc4b5a17bd38d719"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.523121 16393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:08.523824 16408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.523970 16393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:08.524060 16393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "6dcf59e9775648a2b50b487741ee7dfa"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "05d9bca5a76bb14782fc5d2a581be53f"
server_key_iv: "af6da17732acf07cbc4b5a17bd38d719"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.524117 16393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:08.552176 16393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:08.554352 16393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:08.554492 16393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:08.554718 16393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:08.555013 16393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:08.555047 16393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.555074 16393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:08.555097 16393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.560426 16393 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:34829
I20260926 22:49:08.560487 16521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:34829 every 8 connection(s)
I20260926 22:49:08.560794 16393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:08.561184 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 16393
I20260926 22:49:08.561276 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:08.561463 13904 external_mini_cluster.cc:1428] Setting key 2ff3968f8d419b6da8d677007231cf15
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260926 22:49:08.570406 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:08.561541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:35163 (local address 127.13.148.62:34797)
0926 22:49:08.561589 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:08.561591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:08.561885 (+ 294us) server_negotiation.cc:407] Connection header received
0926 22:49:08.562454 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:08.562455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:08.562466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:08.562523 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:08.563512 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.564018 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.565370 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.565583 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.565633 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:08.567886 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:08.567888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:08.567888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:08.567896 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:08.569278 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:08.569648 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:08.569649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:08.569650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:08.569697 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:08.569906 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:08.569906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:08.569907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:08.570004 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:08.570051 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:08.570249 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 22:49:08.570312 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:08.570611 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:08.561529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:34797 (local address 127.13.148.1:35163)
0926 22:49:08.561710 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:08.561778 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:08.562357 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:08.562576 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:08.562579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:08.563134 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:08.563401 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:08.563408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.564100 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.564101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:08.565273 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:08.565276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.565657 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.565658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:08.565829 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:08.566327 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:08.566338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:08.567799 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:08.569356 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:08.569363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:08.569365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:08.569580 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:08.569770 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:08.569770 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:08.569770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:08.569862 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:08.570048 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:08.570050 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:08.570157 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 22:49:08.570233 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 22:49:08.570385 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260926 22:49:08.571069 16522 heartbeater.cc:347] Connected to a master server at 127.13.148.62:34797
I20260926 22:49:08.571157 16522 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:08.571348 16522 heartbeater.cc:511] Master 127.13.148.62:34797 requested a full tablet report, sending...
I20260926 22:49:08.571820 16331 ts_manager.cc:194] Registered new tserver with Master: 6dcf59e9775648a2b50b487741ee7dfa (127.13.148.1:34829)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
I20260926 22:49:08.572404 16331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:35163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:08.584208 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34797
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:08.662302 16527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:08.662483 16527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:08.662511 16527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:08.663601 16527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:08.663640 16527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:08.663672 16527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:08.665021 16527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:08.665236 16527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:08.665438 16527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:08.669967 16532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:08.670053 16527 server_base.cc:1034] running on GCE node
W20260926 22:49:08.670006 16533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:08.669968 16535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:08.670367 16527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:08.670555 16527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:08.671676 16527 hybrid_clock.cc:648] HybridClock initialized: now 1790462948671668 us; error 26 us; skew 500 ppm
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:08.673841 16527 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:08.674238 16527 webserver.cc:460] Webserver started at http://127.13.148.2:46717/ using document root <none> and password file <none>
I20260926 22:49:08.674456 16527 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:08.674501 16527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:08.674619 16527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:08.675606 16527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "380f6adbbdc44675abf88a89c3e7b7d3"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "3627c28a0e2f392a0c55327fcb54dbe5"
server_key_iv: "157b3298e9ecf156a8a6f9fdec25a417"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.675913 16527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "380f6adbbdc44675abf88a89c3e7b7d3"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "3627c28a0e2f392a0c55327fcb54dbe5"
server_key_iv: "157b3298e9ecf156a8a6f9fdec25a417"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.677127 16527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:08.677839 16542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.677999 16527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.678074 16527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "380f6adbbdc44675abf88a89c3e7b7d3"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "3627c28a0e2f392a0c55327fcb54dbe5"
server_key_iv: "157b3298e9ecf156a8a6f9fdec25a417"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.678139 16527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:08.716454 16527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:08.718212 16527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:08.718351 16527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:08.718559 16527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:08.718849 16527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:08.718883 16527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.718911 16527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:08.718935 16527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.724298 16527 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:42017
I20260926 22:49:08.724345 16655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:42017 every 8 connection(s)
I20260926 22:49:08.724637 16527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:08.728613 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 16527
I20260926 22:49:08.728710 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:08.728880 13904 external_mini_cluster.cc:1428] Setting key 1c0de8a024051300267f1855e17ef1cf
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:08.733624 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:08.725276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:35573 (local address 127.13.148.62:34797)
0926 22:49:08.725337 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:08.725339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:08.725552 (+ 213us) server_negotiation.cc:407] Connection header received
0926 22:49:08.726083 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:08.726084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:08.726096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:08.726147 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:08.727129 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.727626 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.728631 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.728915 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.728969 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:08.731213 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:08.731215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:08.731216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:08.731223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:08.732411 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:08.732758 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:08.732760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:08.732760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:08.732813 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:08.733022 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:08.733023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:08.733023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:08.733137 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:08.733190 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:08.733459 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 22:49:08.733534 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:49:08.733842 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:08.725281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:34797 (local address 127.13.148.2:35573)
0926 22:49:08.725438 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:08.725474 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:08.725990 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:08.726218 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:08.726221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:08.726822 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:08.727044 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:08.727050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.727708 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.727709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:08.728552 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:08.728555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.728997 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.728998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:08.729145 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:08.729627 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:08.729636 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:08.731126 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:08.732479 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:08.732486 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:08.732488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:08.732696 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:08.732864 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:08.732865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:08.732865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:08.732970 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:08.733190 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:08.733197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:08.733340 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 22:49:08.733430 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 22:49:08.733585 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260926 22:49:08.734261 16656 heartbeater.cc:347] Connected to a master server at 127.13.148.62:34797
I20260926 22:49:08.734366 16656 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:08.734534 16656 heartbeater.cc:511] Master 127.13.148.62:34797 requested a full tablet report, sending...
I20260926 22:49:08.734881 16331 ts_manager.cc:194] Registered new tserver with Master: 380f6adbbdc44675abf88a89c3e7b7d3 (127.13.148.2:42017)
I20260926 22:49:08.735383 16331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:35573
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:08.752008 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34797
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:08.829917 16661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:08.830106 16661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:08.830127 16661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:08.831218 16661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:08.831252 16661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:08.831275 16661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:08.832626 16661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:08.832803 16661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:08.832974 16661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:08.837394 16669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:08.837409 16667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:08.837558 16661 server_base.cc:1034] running on GCE node
W20260926 22:49:08.837405 16666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:08.837855 16661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:08.838074 16661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:08.839217 16661 hybrid_clock.cc:648] HybridClock initialized: now 1790462948839194 us; error 33 us; skew 500 ppm
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:08.841408 16661 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:08.841907 16661 webserver.cc:460] Webserver started at http://127.13.148.3:45199/ using document root <none> and password file <none>
I20260926 22:49:08.842108 16661 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:08.842149 16661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:08.842238 16661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:08.843152 16661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "687048d6285d4188937ff2d1d1d5a12c"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "cd81756ecd988640f0821a3b1e4a6e9e"
server_key_iv: "974dc362c83772b1d0b5f640760d055e"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.843430 16661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "687048d6285d4188937ff2d1d1d5a12c"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "cd81756ecd988640f0821a3b1e4a6e9e"
server_key_iv: "974dc362c83772b1d0b5f640760d055e"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.844524 16661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:49:08.845119 16676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.845254 16661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:08.845333 16661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "687048d6285d4188937ff2d1d1d5a12c"
format_stamp: "Formatted at 2026-09-26 22:49:08 on dist-test-slave-98hb"
server_key: "cd81756ecd988640f0821a3b1e4a6e9e"
server_key_iv: "974dc362c83772b1d0b5f640760d055e"
server_key_version: "encryptionkey@0"
I20260926 22:49:08.845397 16661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:08.863013 16661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:08.864746 16661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:08.864867 16661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:08.865054 16661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:08.865339 16661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:08.865373 16661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.865393 16661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:08.865404 16661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:08.870916 16661 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:34383
I20260926 22:49:08.870975 16789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:34383 every 8 connection(s)
I20260926 22:49:08.871251 16661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:08.876488 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 16661
I20260926 22:49:08.876596 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:08.876818 13904 external_mini_cluster.cc:1428] Setting key e7ab5f44e7b2ac6adaa83011346044b4
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:08.879575 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:08.871855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:46911 (local address 127.13.148.62:34797)
0926 22:49:08.871898 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:08.871900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:08.872128 (+ 228us) server_negotiation.cc:407] Connection header received
0926 22:49:08.872577 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:08.872578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:08.872587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:08.872637 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:08.873483 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.874031 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.874967 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.875177 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.875224 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:08.877508 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:08.877509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:08.877510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:08.877516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:08.878584 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:08.878908 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:08.878909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:08.878909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:08.878956 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:08.879133 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:08.879133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:08.879134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:08.879224 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:08.879267 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:08.879463 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 22:49:08.879515 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:49:08.879741 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:08.871853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:34797 (local address 127.13.148.3:46911)
0926 22:49:08.872011 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:08.872047 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:08.872502 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:08.872691 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:08.872694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:08.873224 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:08.873410 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:08.873415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.874101 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.874102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:08.874892 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:08.874895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:08.875234 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:08.875234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:08.875371 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:08.875804 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:08.875812 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:08.877326 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:08.878645 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:08.878650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:08.878651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:08.878854 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:08.879000 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:08.879001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:08.879001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:08.879090 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:08.879265 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:08.879267 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:08.879377 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 22:49:08.879453 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 22:49:08.879560 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260926 22:49:08.880141 16790 heartbeater.cc:347] Connected to a master server at 127.13.148.62:34797
I20260926 22:49:08.880219 16790 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:08.880378 16790 heartbeater.cc:511] Master 127.13.148.62:34797 requested a full tablet report, sending...
I20260926 22:49:08.880692 16331 ts_manager.cc:194] Registered new tserver with Master: 687048d6285d4188937ff2d1d1d5a12c (127.13.148.3:34383)
I20260926 22:49:08.881033 16331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:46911
I20260926 22:49:08.888337 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 22:49:08.898598 16371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:08.897618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.13.148.62:34797)
0926 22:49:08.897686 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:08.897688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:08.897793 (+ 105us) server_negotiation.cc:407] Connection header received
0926 22:49:08.897875 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:08.897876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:08.897885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:08.897932 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:08.898521 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45580: BlockingRecv error: recv got EOF from 127.0.0.1:45580 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 22:49:08.899827 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 16393
I20260926 22:49:08.903786 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 16527
I20260926 22:49:08.907626 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 16661
I20260926 22:49:08.911028 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 16299
2026-09-26T22:49:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1705 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16802](info): set up 2 sockets
Sep 26 22:49:08 dist-test-slave-98hb krb5kdc[16802](info): commencing operation
krb5kdc: starting...
W20260926 22:49:09.853619 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:09 dist-test-slave-98hb krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:09.962188 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:33335
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:10.040728 16815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:10.040936 16815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:10.040957 16815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:10.042104 16815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:10.042142 16815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:10.042155 16815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:10.042165 16815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:10.042181 16815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:10.043617 16815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:10.043804 16815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:10.043973 16815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:10.048484 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:10.048471 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:10.048547 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:10.048658 16815 server_base.cc:1034] running on GCE node
I20260926 22:49:10.048898 16815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:10.049155 16815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:10.050278 16815 hybrid_clock.cc:648] HybridClock initialized: now 1790462950050263 us; error 31 us; skew 500 ppm
Sep 26 22:49:10 dist-test-slave-98hb krb5kdc[16802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462950, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:10.052464 16815 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:10.052870 16815 webserver.cc:460] Webserver started at http://127.13.148.62:42759/ using document root <none> and password file <none>
I20260926 22:49:10.053057 16815 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:10.053103 16815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:10.053232 16815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:10.054301 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "28319c6e77f9424a98b36aa123700689"
format_stamp: "Formatted at 2026-09-26 22:49:10 on dist-test-slave-98hb"
server_key: "403fffa4b32039cfaeb10b37e8a03409"
server_key_iv: "12ece47ee8a5d8f2eb850525be300018"
server_key_version: "encryptionkey@0"
I20260926 22:49:10.054616 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "28319c6e77f9424a98b36aa123700689"
format_stamp: "Formatted at 2026-09-26 22:49:10 on dist-test-slave-98hb"
server_key: "403fffa4b32039cfaeb10b37e8a03409"
server_key_iv: "12ece47ee8a5d8f2eb850525be300018"
server_key_version: "encryptionkey@0"
I20260926 22:49:10.055804 16815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:10.056458 16830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:10.056603 16815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:10.056684 16815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "28319c6e77f9424a98b36aa123700689"
format_stamp: "Formatted at 2026-09-26 22:49:10 on dist-test-slave-98hb"
server_key: "403fffa4b32039cfaeb10b37e8a03409"
server_key_iv: "12ece47ee8a5d8f2eb850525be300018"
server_key_version: "encryptionkey@0"
I20260926 22:49:10.056741 16815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:10.072996 16815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:10.074821 16815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:10.074962 16815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:10.078711 16815 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:33335
I20260926 22:49:10.078747 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:33335 every 8 connection(s)
I20260926 22:49:10.079195 16815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:10.079802 16883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:10.082031 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689: Bootstrap starting.
I20260926 22:49:10.082581 16883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:10.082844 16883 log.cc:824] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:10.083451 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689: No bootstrap required, opened a new log
I20260926 22:49:10.084939 16883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28319c6e77f9424a98b36aa123700689" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33335 } }
I20260926 22:49:10.085071 16883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:10.085103 16883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28319c6e77f9424a98b36aa123700689, State: Initialized, Role: FOLLOWER
I20260926 22:49:10.085242 16883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28319c6e77f9424a98b36aa123700689" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33335 } }
I20260926 22:49:10.085306 16883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:10.085338 16883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:10.085383 16883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:10.086159 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 16815
I20260926 22:49:10.086254 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:10.086347 16883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28319c6e77f9424a98b36aa123700689" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33335 } }
I20260926 22:49:10.086424 13904 external_mini_cluster.cc:1428] Setting key 6a15d58e990a13e5849b211dc28a1e23
I20260926 22:49:10.086447 16883 leader_election.cc:302] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28319c6e77f9424a98b36aa123700689; no voters:
I20260926 22:49:10.086603 16883 leader_election.cc:288] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:10.086642 16886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:10.086810 16886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [term 1 LEADER]: Becoming Leader. State: Replica: 28319c6e77f9424a98b36aa123700689, State: Running, Role: LEADER
I20260926 22:49:10.086855 16883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:10.086966 16886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28319c6e77f9424a98b36aa123700689" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33335 } }
I20260926 22:49:10.087357 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28319c6e77f9424a98b36aa123700689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28319c6e77f9424a98b36aa123700689" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33335 } } }
I20260926 22:49:10.087442 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:10.087347 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28319c6e77f9424a98b36aa123700689. Latest consensus state: current_term: 1 leader_uuid: "28319c6e77f9424a98b36aa123700689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28319c6e77f9424a98b36aa123700689" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33335 } } }
I20260926 22:49:10.087647 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:10.087967 16897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:10.088517 16897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:10.089854 16897 catalog_manager.cc:1347] Generated new cluster ID: f22dc76f32e64a81882d1a41b52cc634
I20260926 22:49:10.089902 16897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:49:10 dist-test-slave-98hb krb5kdc[16802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:10.096539 16897 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:10.097169 16897 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:10.097911 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:10.087010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.13.148.62:33335)
0926 22:49:10.087164 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:10.087174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:10.087239 (+ 65us) server_negotiation.cc:407] Connection header received
0926 22:49:10.087544 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:10.087552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:10.087612 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:10.087753 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:10.088611 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:10.089997 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:10.091038 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:10.091673 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:10.091779 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:10.094436 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:10.094440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:10.094440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:10.094453 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:10.096315 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:10.096698 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:10.096702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:10.096702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:10.096807 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:10.097003 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:10.097004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:10.097005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:10.097275 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:10.097420 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:10.097516 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 22:49:10.097655 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260926 22:49:10.098762 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:49:10.104825 16897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28319c6e77f9424a98b36aa123700689: Generated new TSK 0
I20260926 22:49:10.104993 16897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:49:10.113345 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:10.109692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.13.148.62:33335)
0926 22:49:10.109774 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:10.109776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:10.109881 (+ 105us) server_negotiation.cc:407] Connection header received
0926 22:49:10.109954 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:10.109955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:10.109964 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:10.110012 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:10.110440 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:10.110946 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:10.112004 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:10.112229 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:10.112276 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:10.112936 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:10.112937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:10.112938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:10.113124 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:10.113140 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:10.113146 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:10.113186 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:10.113258 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 22:49:10.113284 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:49:10.115866 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 16815
2026-09-26T22:49:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1203 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:10 dist-test-slave-98hb krb5kdc[16916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:10 dist-test-slave-98hb krb5kdc[16916](info): set up 2 sockets
Sep 26 22:49:10 dist-test-slave-98hb krb5kdc[16916](info): commencing operation
krb5kdc: starting...
W20260926 22:49:11.061584 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:11.172968 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39535
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:11.252102 16929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:11.252326 16929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:11.252349 16929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:11.253461 16929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:11.253494 16929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:11.253506 16929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:11.253517 16929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:11.253528 16929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:11.255012 16929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:11.255209 16929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:11.255383 16929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:11.259959 16935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:11.259969 16937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:11.259991 16934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:11.260088 16929 server_base.cc:1034] running on GCE node
I20260926 22:49:11.260502 16929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:11.260788 16929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:11.261924 16929 hybrid_clock.cc:648] HybridClock initialized: now 1790462951261908 us; error 28 us; skew 500 ppm
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:11.264171 16929 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:11.264585 16929 webserver.cc:460] Webserver started at http://127.13.148.62:33245/ using document root <none> and password file <none>
I20260926 22:49:11.264791 16929 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:11.264837 16929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:11.264961 16929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:11.266024 16929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "65607d5bd05645f3a62917840fb8e46b"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "49dde3ffb8748215ee72f5a6d0d7d020"
server_key_iv: "b252b1f1a84efa4d52c231f0e16ebe85"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.266368 16929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "65607d5bd05645f3a62917840fb8e46b"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "49dde3ffb8748215ee72f5a6d0d7d020"
server_key_iv: "b252b1f1a84efa4d52c231f0e16ebe85"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.267505 16929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:11.268113 16944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.268266 16929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:11.268349 16929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "65607d5bd05645f3a62917840fb8e46b"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "49dde3ffb8748215ee72f5a6d0d7d020"
server_key_iv: "b252b1f1a84efa4d52c231f0e16ebe85"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.268404 16929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:11.281237 16929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:11.283277 16929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:11.283401 16929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:11.287065 16929 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39535
I20260926 22:49:11.287098 16996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39535 every 8 connection(s)
I20260926 22:49:11.287405 16929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:11.288301 16997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:11.290467 16997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b: Bootstrap starting.
I20260926 22:49:11.290980 16997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:11.291199 16997 log.cc:824] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:11.291893 16997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b: No bootstrap required, opened a new log
I20260926 22:49:11.293467 16997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65607d5bd05645f3a62917840fb8e46b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39535 } }
I20260926 22:49:11.293609 16997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:11.293630 16997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65607d5bd05645f3a62917840fb8e46b, State: Initialized, Role: FOLLOWER
I20260926 22:49:11.293771 16997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65607d5bd05645f3a62917840fb8e46b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39535 } }
I20260926 22:49:11.293846 16997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:11.293869 16997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:11.293901 16997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:11.294673 16997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65607d5bd05645f3a62917840fb8e46b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39535 } }
I20260926 22:49:11.294767 16997 leader_election.cc:302] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65607d5bd05645f3a62917840fb8e46b; no voters:
I20260926 22:49:11.294895 16997 leader_election.cc:288] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:11.294934 17000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:11.295123 16997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:11.295135 17000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [term 1 LEADER]: Becoming Leader. State: Replica: 65607d5bd05645f3a62917840fb8e46b, State: Running, Role: LEADER
I20260926 22:49:11.295286 17000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65607d5bd05645f3a62917840fb8e46b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39535 } }
I20260926 22:49:11.295687 17001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65607d5bd05645f3a62917840fb8e46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65607d5bd05645f3a62917840fb8e46b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39535 } } }
I20260926 22:49:11.295783 17001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:11.296000 17009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:11.295992 17002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65607d5bd05645f3a62917840fb8e46b. Latest consensus state: current_term: 1 leader_uuid: "65607d5bd05645f3a62917840fb8e46b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65607d5bd05645f3a62917840fb8e46b" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39535 } } }
I20260926 22:49:11.296368 17002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:11.296689 17009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:11.297520 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 16929
I20260926 22:49:11.297619 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:11.297829 13904 external_mini_cluster.cc:1428] Setting key 63f7c9d5925ea83fc458df8cfafdfa0a
I20260926 22:49:11.298492 17009 catalog_manager.cc:1347] Generated new cluster ID: 09fa0dde6bc84720a8921e4c8901d060
I20260926 22:49:11.298542 17009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:11.312326 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:11.298288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.13.148.62:39535)
0926 22:49:11.301928 (+ 3640us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:11.301937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:11.301970 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:49:11.302083 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:11.302090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:11.302144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:11.302279 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:11.303044 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.304418 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.305489 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.306104 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.306194 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:11.308779 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:11.308784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:11.308784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:11.308796 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:11.310673 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.311030 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.311033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.311034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.311142 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.311308 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.311308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.311309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.311615 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:11.311756 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:11.311850 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 22:49:11.312090 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3601,"thread_start_us":68,"threads_started":1}
I20260926 22:49:11.314788 17009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:11.315464 17009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:11.322836 17009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65607d5bd05645f3a62917840fb8e46b: Generated new TSK 0
I20260926 22:49:11.323009 17009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:11.348984 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39535
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:11.426944 17023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:11.427135 17023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:11.427156 17023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:11.428222 17023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:11.428256 17023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:11.428284 17023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:11.429629 17023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:11.429838 17023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:11.430018 17023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:11.434536 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:11.434582 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:11.434551 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:11.434693 17023 server_base.cc:1034] running on GCE node
I20260926 22:49:11.434934 17023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:11.435153 17023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:11.436326 17023 hybrid_clock.cc:648] HybridClock initialized: now 1790462951436311 us; error 28 us; skew 500 ppm
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:11.438747 17023 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:11.439181 17023 webserver.cc:460] Webserver started at http://127.13.148.1:38531/ using document root <none> and password file <none>
I20260926 22:49:11.439388 17023 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:11.439435 17023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:11.439555 17023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:11.440563 17023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "d0376bbdb16f4cbcb95db21d18f903a3"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "537b7871bce4de7f0bb8727ea7aeb4f4"
server_key_iv: "42d0912e0062b0eeca37924d004dccf2"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.440877 17023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "d0376bbdb16f4cbcb95db21d18f903a3"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "537b7871bce4de7f0bb8727ea7aeb4f4"
server_key_iv: "42d0912e0062b0eeca37924d004dccf2"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.442103 17023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:11.442770 17038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.442915 17023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:11.443017 17023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "d0376bbdb16f4cbcb95db21d18f903a3"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "537b7871bce4de7f0bb8727ea7aeb4f4"
server_key_iv: "42d0912e0062b0eeca37924d004dccf2"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.443075 17023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:11.456686 17023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:11.458616 17023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:11.458741 17023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:11.458947 17023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:11.459229 17023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:11.459261 17023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.459290 17023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:11.459312 17023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.464530 17023 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:38863
I20260926 22:49:11.464588 17151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:38863 every 8 connection(s)
I20260926 22:49:11.464883 17023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:11.473232 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:11.465519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42327 (local address 127.13.148.62:39535)
0926 22:49:11.465580 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:11.465582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:11.465747 (+ 165us) server_negotiation.cc:407] Connection header received
0926 22:49:11.466263 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:11.466264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:11.466276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:11.466332 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:11.467236 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.467738 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.468689 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.468904 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.468956 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:11.470938 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:11.470940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:11.470940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:11.470947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:11.472060 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.472403 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.472404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.472404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.472459 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.472658 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.472659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.472659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.472766 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:11.472821 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:11.473063 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 22:49:11.473144 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:49:11.473409 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:11.465484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39535 (local address 127.13.148.1:42327)
0926 22:49:11.465636 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:11.465670 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:11.466169 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:11.466402 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:11.466406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:11.466961 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:11.467143 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:11.467147 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.467819 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.467820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:11.468609 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:11.468610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.468967 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.468968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:11.469086 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:11.469494 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:11.469502 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:11.470863 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:11.472127 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:11.472133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:11.472135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:11.472340 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:11.472511 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:11.472512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:11.472512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:11.472607 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:11.472819 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:11.472825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:11.472954 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 22:49:11.473037 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:49:11.473182 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260926 22:49:11.473491 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 17023
I20260926 22:49:11.473590 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:11.473789 13904 external_mini_cluster.cc:1428] Setting key 7951525b96cef455219258548d849ede
I20260926 22:49:11.473874 17152 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39535
I20260926 22:49:11.473975 17152 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:11.474165 17152 heartbeater.cc:511] Master 127.13.148.62:39535 requested a full tablet report, sending...
I20260926 22:49:11.474668 16961 ts_manager.cc:194] Registered new tserver with Master: d0376bbdb16f4cbcb95db21d18f903a3 (127.13.148.1:38863)
I20260926 22:49:11.475256 16961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:42327
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:11.495980 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39535
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:11.575382 17157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:11.575587 17157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:11.575619 17157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:11.576776 17157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:11.576817 17157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:11.576853 17157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:11.578346 17157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:11.578588 17157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:11.578794 17157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:11.583433 17163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:11.583516 17157 server_base.cc:1034] running on GCE node
W20260926 22:49:11.583472 17165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:11.583437 17162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:11.583787 17157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:11.584007 17157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:11.585140 17157 hybrid_clock.cc:648] HybridClock initialized: now 1790462951585128 us; error 29 us; skew 500 ppm
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:11.587361 17157 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:11.587764 17157 webserver.cc:460] Webserver started at http://127.13.148.2:46579/ using document root <none> and password file <none>
I20260926 22:49:11.587962 17157 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:11.588008 17157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:11.588130 17157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:11.589123 17157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "4c6bfb5d88ff491b95b43df9d5fdd10a"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "f4b84a7ee6c88e9d7f7daa22e652c18e"
server_key_iv: "7d7ab3d86843666df4c8e35f854e105c"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.589442 17157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "4c6bfb5d88ff491b95b43df9d5fdd10a"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "f4b84a7ee6c88e9d7f7daa22e652c18e"
server_key_iv: "7d7ab3d86843666df4c8e35f854e105c"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.590754 17157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:11.591401 17172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.591580 17157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:11.591658 17157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "4c6bfb5d88ff491b95b43df9d5fdd10a"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "f4b84a7ee6c88e9d7f7daa22e652c18e"
server_key_iv: "7d7ab3d86843666df4c8e35f854e105c"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.591730 17157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:11.606288 17157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:11.607841 17157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:11.607965 17157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:11.608161 17157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:11.608433 17157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:11.608464 17157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.608491 17157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:11.608513 17157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.613713 17157 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:45299
I20260926 22:49:11.613811 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:45299 every 8 connection(s)
I20260926 22:49:11.614048 17157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:11.620786 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 17157
I20260926 22:49:11.620870 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:11.621035 13904 external_mini_cluster.cc:1428] Setting key de926054cce2a4b755578008cc78eba4
I20260926 22:49:11.622206 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:11.614639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:46027 (local address 127.13.148.62:39535)
0926 22:49:11.614721 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:11.614723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:11.614878 (+ 155us) server_negotiation.cc:407] Connection header received
0926 22:49:11.615353 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:11.615354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:11.615363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:11.615410 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:11.616242 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.616719 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.617607 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.617845 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.617893 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:11.619681 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:11.619683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:11.619683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:11.619689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:11.620919 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.621318 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.621320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.621320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.621376 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.621599 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.621600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.621600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.621709 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:11.621808 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:11.622042 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 22:49:11.622121 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 22:49:11.622375 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:11.614613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39535 (local address 127.13.148.2:46027)
0926 22:49:11.614754 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:11.614793 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:11.615276 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:11.615462 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:11.615465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:11.615981 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:11.616176 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:11.616180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.616785 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.616786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:11.617549 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:11.617551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.617902 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.617903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:11.618003 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:11.618386 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:11.618394 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:11.619604 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:11.620986 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:11.620992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:11.620995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:11.621254 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:11.621428 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:11.621429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:11.621429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:11.621538 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:11.621788 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:11.621795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:11.621929 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 22:49:11.622016 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 22:49:11.622160 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260926 22:49:11.623255 17286 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39535
I20260926 22:49:11.623354 17286 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:11.623523 17286 heartbeater.cc:511] Master 127.13.148.62:39535 requested a full tablet report, sending...
I20260926 22:49:11.623853 16961 ts_manager.cc:194] Registered new tserver with Master: 4c6bfb5d88ff491b95b43df9d5fdd10a (127.13.148.2:45299)
I20260926 22:49:11.624308 16961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:46027
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:11.644464 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39535
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:11.723457 17291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:11.723687 17291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:11.723711 17291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:11.724825 17291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:11.724861 17291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:11.724885 17291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:11.726342 17291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:11.726550 17291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:11.726742 17291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:11.731478 17299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:11.731586 17297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:11.731495 17296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:11.731519 17291 server_base.cc:1034] running on GCE node
I20260926 22:49:11.731912 17291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:11.732192 17291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:11.733357 17291 hybrid_clock.cc:648] HybridClock initialized: now 1790462951733342 us; error 30 us; skew 500 ppm
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:11.735831 17291 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:11.736369 17291 webserver.cc:460] Webserver started at http://127.13.148.3:34485/ using document root <none> and password file <none>
I20260926 22:49:11.736610 17291 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:11.736660 17291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:11.736797 17291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:11.737897 17291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "6424fb68e551454789f0a390f2d25fb9"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "5414e9b4e382a0602a183dd81087583f"
server_key_iv: "31067b66e26dbe1cd97ba2061c4a8c8f"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.738241 17291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "6424fb68e551454789f0a390f2d25fb9"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "5414e9b4e382a0602a183dd81087583f"
server_key_iv: "31067b66e26dbe1cd97ba2061c4a8c8f"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.739527 17291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:11.740345 17306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.740510 17291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:11.740593 17291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "6424fb68e551454789f0a390f2d25fb9"
format_stamp: "Formatted at 2026-09-26 22:49:11 on dist-test-slave-98hb"
server_key: "5414e9b4e382a0602a183dd81087583f"
server_key_iv: "31067b66e26dbe1cd97ba2061c4a8c8f"
server_key_version: "encryptionkey@0"
I20260926 22:49:11.740654 17291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:11.752676 17291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:11.754835 17291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:11.754976 17291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:11.755194 17291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:11.755494 17291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:11.755527 17291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.755555 17291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:11.755579 17291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:11.760998 17291 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:40123
I20260926 22:49:11.761035 17419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:40123 every 8 connection(s)
I20260926 22:49:11.761345 17291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:11.768899 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 17291
I20260926 22:49:11.769006 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:11.769189 13904 external_mini_cluster.cc:1428] Setting key 7e3ec39ec9a88a4a003217f23aad7215
I20260926 22:49:11.769542 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:11.762049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:59411 (local address 127.13.148.62:39535)
0926 22:49:11.762101 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:11.762103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:11.762290 (+ 187us) server_negotiation.cc:407] Connection header received
0926 22:49:11.762759 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:11.762760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:11.762769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:11.762833 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:11.763694 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.764177 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.765101 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.765272 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.765313 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:11.767272 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:11.767273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:11.767273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:11.767280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:11.768313 (+ 1033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.768648 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.768648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.768649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.768689 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.768937 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.768937 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.768937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.769050 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:11.769110 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:11.769381 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 22:49:11.769456 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:49:11.769721 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:11.762026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39535 (local address 127.13.148.3:59411)
0926 22:49:11.762170 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:11.762211 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:11.762686 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:11.762888 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:11.762892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:11.763428 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:11.763619 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:11.763623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.764237 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.764237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:11.765040 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:11.765043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.765357 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.765359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:11.765480 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:11.765911 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:11.765919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:11.767164 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:11.768379 (+ 1215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:11.768385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:11.768387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:11.768591 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:11.768741 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:11.768742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:11.768743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:11.768882 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:11.769103 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:11.769110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:11.769244 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 22:49:11.769357 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 22:49:11.769508 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260926 22:49:11.770151 17420 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39535
I20260926 22:49:11.770246 17420 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:11.770409 17420 heartbeater.cc:511] Master 127.13.148.62:39535 requested a full tablet report, sending...
I20260926 22:49:11.770709 16961 ts_manager.cc:194] Registered new tserver with Master: 6424fb68e551454789f0a390f2d25fb9 (127.13.148.3:40123)
I20260926 22:49:11.771029 16961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:59411
I20260926 22:49:11.780762 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[16916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462951, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:11.802078 17020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:11.795836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.13.148.62:39535)
0926 22:49:11.795890 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:11.795892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:11.795903 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:49:11.795960 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:11.795961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:11.795969 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:11.796027 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:11.796609 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.797142 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.797881 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:11.798061 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:11.798130 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:11.800089 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:11.800090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:11.800091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:11.800097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:11.801215 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.801476 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.801477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.801477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.801518 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:11.801632 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:11.801632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:11.801632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:11.801802 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:11.801881 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:11.801957 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 22:49:11.802015 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 22:49:11.802304 16961 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36640
I20260926 22:49:11.802733 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 17023
I20260926 22:49:11.806995 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 17157
I20260926 22:49:11.810933 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 17291
I20260926 22:49:11.814334 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 16929
2026-09-26T22:49:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1699 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[17432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[17432](info): set up 2 sockets
Sep 26 22:49:11 dist-test-slave-98hb krb5kdc[17432](info): commencing operation
krb5kdc: starting...
W20260926 22:49:12.769935 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:12 dist-test-slave-98hb krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:12.880961 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:38259
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:38259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:12.958989 17445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:12.959197 17445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:12.959218 17445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:12.960285 17445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:12.960317 17445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:12.960330 17445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:12.960340 17445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:12.960353 17445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:12.961797 17445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:38259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:38259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:12.961985 17445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:12.962162 17445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:12.966629 17453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:12.966718 17451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:12.966615 17450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:12.966812 17445 server_base.cc:1034] running on GCE node
I20260926 22:49:12.967029 17445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:12.967309 17445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:12.968462 17445 hybrid_clock.cc:648] HybridClock initialized: now 1790462952968448 us; error 27 us; skew 500 ppm
Sep 26 22:49:12 dist-test-slave-98hb krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462952, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:12.970863 17445 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:12.971282 17445 webserver.cc:460] Webserver started at http://127.13.148.62:43897/ using document root <none> and password file <none>
I20260926 22:49:12.971472 17445 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:12.971514 17445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:12.971634 17445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:12.972676 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "79573b6c07ef476bacce18633e6e142c"
format_stamp: "Formatted at 2026-09-26 22:49:12 on dist-test-slave-98hb"
server_key: "065bdd10252ec9a38278769e8dc38732"
server_key_iv: "76fa9d8505bd0ed57aaef1028ccd361c"
server_key_version: "encryptionkey@0"
I20260926 22:49:12.972995 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "79573b6c07ef476bacce18633e6e142c"
format_stamp: "Formatted at 2026-09-26 22:49:12 on dist-test-slave-98hb"
server_key: "065bdd10252ec9a38278769e8dc38732"
server_key_iv: "76fa9d8505bd0ed57aaef1028ccd361c"
server_key_version: "encryptionkey@0"
I20260926 22:49:12.974295 17445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:12.974992 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:12.975162 17445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:12.975242 17445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "79573b6c07ef476bacce18633e6e142c"
format_stamp: "Formatted at 2026-09-26 22:49:12 on dist-test-slave-98hb"
server_key: "065bdd10252ec9a38278769e8dc38732"
server_key_iv: "76fa9d8505bd0ed57aaef1028ccd361c"
server_key_version: "encryptionkey@0"
I20260926 22:49:12.975317 17445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:12.995503 17445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:12.997303 17445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:12.997429 17445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:13.001281 17445 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:38259
I20260926 22:49:13.001315 17512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:38259 every 8 connection(s)
I20260926 22:49:13.001809 17445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:13.002440 17513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:13.004644 17513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c: Bootstrap starting.
I20260926 22:49:13.005177 17513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:13.005412 17513 log.cc:824] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:13.005671 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 17445
I20260926 22:49:13.005797 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:13.005962 13904 external_mini_cluster.cc:1428] Setting key 2c71f73a0f04e389a8525cb4a7e9ad18
I20260926 22:49:13.006101 17513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c: No bootstrap required, opened a new log
I20260926 22:49:13.008652 17513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79573b6c07ef476bacce18633e6e142c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38259 } }
I20260926 22:49:13.008829 17513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:13.008860 17513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79573b6c07ef476bacce18633e6e142c, State: Initialized, Role: FOLLOWER
I20260926 22:49:13.008980 17513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79573b6c07ef476bacce18633e6e142c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38259 } }
I20260926 22:49:13.009047 17513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:13.009083 17513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:13.009121 17513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:13.009706 17513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79573b6c07ef476bacce18633e6e142c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38259 } }
I20260926 22:49:13.009835 17513 leader_election.cc:302] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79573b6c07ef476bacce18633e6e142c; no voters:
I20260926 22:49:13.009989 17513 leader_election.cc:288] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:13.010027 17518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:13.010210 17518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [term 1 LEADER]: Becoming Leader. State: Replica: 79573b6c07ef476bacce18633e6e142c, State: Running, Role: LEADER
I20260926 22:49:13.010231 17513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:13.010371 17518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79573b6c07ef476bacce18633e6e142c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38259 } }
I20260926 22:49:13.011559 17520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79573b6c07ef476bacce18633e6e142c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79573b6c07ef476bacce18633e6e142c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38259 } } }
I20260926 22:49:13.011607 17519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79573b6c07ef476bacce18633e6e142c. Latest consensus state: current_term: 1 leader_uuid: "79573b6c07ef476bacce18633e6e142c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79573b6c07ef476bacce18633e6e142c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 38259 } } }
I20260926 22:49:13.011899 17520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:13.011929 17519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c [sys.catalog]: This master's current role is: LEADER
W20260926 22:49:13.012174 17533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:49:13.012224 17533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:49:13.012306 17535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:13.012487 17535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:13.014390 17535 catalog_manager.cc:1347] Generated new cluster ID: 907fa93c1ce74ddd9bb49f33e7de9619
I20260926 22:49:13.014442 17535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:13.017680 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.006436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.13.148.62:38259)
0926 22:49:13.006600 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:13.006610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:13.006645 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:49:13.007346 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:13.007356 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:13.007420 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:13.007563 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:13.008112 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.009444 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.010427 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.010998 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.011080 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.013951 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:13.013955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:13.013955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:13.013967 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:13.015984 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.016377 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.016380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.016380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.016484 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.016658 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.016658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.016659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.016947 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:13.017098 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:13.017179 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 22:49:13.017324 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260926 22:49:13.024060 17535 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:13.024544 17535 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:13.029433 17535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79573b6c07ef476bacce18633e6e142c: Generated new TSK 0
I20260926 22:49:13.029595 17535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:13.060688 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38259
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:13.141124 17539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:13.141330 17539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:13.141360 17539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:13.142531 17539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:13.142570 17539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:13.142613 17539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:13.144078 17539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:13.144305 17539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:13.144507 17539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:13.149021 17547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:13.149032 17544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:13.149204 17539 server_base.cc:1034] running on GCE node
W20260926 22:49:13.149034 17545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:13.149428 17539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:13.149658 17539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:13.150790 17539 hybrid_clock.cc:648] HybridClock initialized: now 1790462953150785 us; error 38 us; skew 500 ppm
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:13.153023 17539 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:13.153411 17539 webserver.cc:460] Webserver started at http://127.13.148.1:35203/ using document root <none> and password file <none>
I20260926 22:49:13.153600 17539 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:13.153644 17539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:13.153787 17539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:13.154776 17539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "16f4265c9afa4fbb93debe4a2b4bf965"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "e88a2e6489f3eff6073102c67b8243f6"
server_key_iv: "631df9fb133db1302f651c05cdc6f58c"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.155088 17539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "16f4265c9afa4fbb93debe4a2b4bf965"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "e88a2e6489f3eff6073102c67b8243f6"
server_key_iv: "631df9fb133db1302f651c05cdc6f58c"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.156323 17539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:13.156998 17554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.157147 17539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:13.157222 17539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "16f4265c9afa4fbb93debe4a2b4bf965"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "e88a2e6489f3eff6073102c67b8243f6"
server_key_iv: "631df9fb133db1302f651c05cdc6f58c"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.157292 17539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:13.165515 17539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:13.167311 17539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:13.167433 17539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:13.167623 17539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:13.167894 17539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:13.167925 17539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.167953 17539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:13.167975 17539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.173205 17539 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:41193
I20260926 22:49:13.173256 17667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:41193 every 8 connection(s)
I20260926 22:49:13.173548 17539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:13.175588 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 17539
I20260926 22:49:13.175710 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:13.175902 13904 external_mini_cluster.cc:1428] Setting key c2a0044ea3d9c5dc2d1b28ec51a869dc
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:13.183085 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.174219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:37269 (local address 127.13.148.62:38259)
0926 22:49:13.174260 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:13.174262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:13.174465 (+ 203us) server_negotiation.cc:407] Connection header received
0926 22:49:13.175344 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:13.175345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:13.175353 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:13.175394 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:13.176203 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.176757 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.177873 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.178105 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.178162 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.180493 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:13.180495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:13.180495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:13.180501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:13.181716 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.182204 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.182205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.182205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.182252 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.182497 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.182497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.182497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.182592 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:13.182634 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:13.182953 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 22:49:13.183018 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:49:13.183353 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.174196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38259 (local address 127.13.148.1:37269)
0926 22:49:13.174349 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:13.174389 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:13.175248 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:13.175499 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:13.175503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:13.175843 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:13.176134 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:13.176140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.176839 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.176840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:13.177802 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:13.177806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.178180 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.178181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:13.178318 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.178793 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:13.178801 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:13.180400 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:13.181834 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:13.181843 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:13.181846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:13.182137 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:13.182305 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:13.182306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:13.182306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:13.182441 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:13.182645 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:13.182650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:13.182827 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 22:49:13.182945 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 22:49:13.183110 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260926 22:49:13.183856 17668 heartbeater.cc:347] Connected to a master server at 127.13.148.62:38259
I20260926 22:49:13.183938 17668 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:13.184105 17668 heartbeater.cc:511] Master 127.13.148.62:38259 requested a full tablet report, sending...
I20260926 22:49:13.184574 17477 ts_manager.cc:194] Registered new tserver with Master: 16f4265c9afa4fbb93debe4a2b4bf965 (127.13.148.1:41193)
I20260926 22:49:13.185178 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:37269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:13.200026 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38259
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:13.278445 17673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:13.278651 17673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:13.278673 17673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:13.279817 17673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:13.279857 17673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:13.279881 17673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:13.281311 17673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:13.281487 17673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:13.281661 17673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:13.286197 17681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:13.286290 17678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:13.286197 17679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:13.286386 17673 server_base.cc:1034] running on GCE node
I20260926 22:49:13.286586 17673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:13.286820 17673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:13.287983 17673 hybrid_clock.cc:648] HybridClock initialized: now 1790462953287968 us; error 29 us; skew 500 ppm
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:13.290383 17673 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:13.290789 17673 webserver.cc:460] Webserver started at http://127.13.148.2:46473/ using document root <none> and password file <none>
I20260926 22:49:13.290977 17673 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:13.291016 17673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:13.291113 17673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:13.292068 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "08023a6b868242d2b60a7e7c1d308fe1"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "8a2a156c278ad3327523bb7d2f3eda15"
server_key_iv: "950d52f6f0f5f661a5fd4d1c3870e38b"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.292356 17673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "08023a6b868242d2b60a7e7c1d308fe1"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "8a2a156c278ad3327523bb7d2f3eda15"
server_key_iv: "950d52f6f0f5f661a5fd4d1c3870e38b"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.293471 17673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:13.294126 17688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.294345 17673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.294423 17673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "08023a6b868242d2b60a7e7c1d308fe1"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "8a2a156c278ad3327523bb7d2f3eda15"
server_key_iv: "950d52f6f0f5f661a5fd4d1c3870e38b"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.294494 17673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:13.322017 17673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:13.323766 17673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:13.323899 17673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:13.324115 17673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:13.324433 17673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:13.324469 17673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.324497 17673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:13.324518 17673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.329670 17673 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:39467
I20260926 22:49:13.329715 17801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:39467 every 8 connection(s)
I20260926 22:49:13.330044 17673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:13.334571 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 17673
I20260926 22:49:13.334682 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:13.334851 13904 external_mini_cluster.cc:1428] Setting key a0003f460da0f9185f0991570514f03f
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:13.339071 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.330693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33033 (local address 127.13.148.62:38259)
0926 22:49:13.330745 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:13.330747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:13.330928 (+ 181us) server_negotiation.cc:407] Connection header received
0926 22:49:13.331806 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:13.331807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:13.331819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:13.331866 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:13.332452 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.332936 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.333933 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.334119 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.334173 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.336526 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:13.336527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:13.336528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:13.336534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:13.337880 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.338242 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.338243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.338243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.338312 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.338501 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.338502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.338502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.338621 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:13.338681 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:13.338902 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 22:49:13.338978 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:49:13.339208 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.330669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38259 (local address 127.13.148.2:33033)
0926 22:49:13.330821 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:13.330855 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:13.331707 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:13.331938 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:13.331942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:13.332188 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:13.332370 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:13.332374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.333011 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.333013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:13.333862 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:13.333865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.334182 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.334183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:13.334305 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.334817 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:13.334824 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:13.336443 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:13.337962 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:13.337968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:13.337969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:13.338178 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:13.338367 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:13.338368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:13.338368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:13.338456 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:13.338672 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:13.338675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:13.338788 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 22:49:13.338865 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 22:49:13.338980 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260926 22:49:13.339625 17802 heartbeater.cc:347] Connected to a master server at 127.13.148.62:38259
I20260926 22:49:13.339722 17802 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:13.339915 17802 heartbeater.cc:511] Master 127.13.148.62:38259 requested a full tablet report, sending...
I20260926 22:49:13.340284 17477 ts_manager.cc:194] Registered new tserver with Master: 08023a6b868242d2b60a7e7c1d308fe1 (127.13.148.2:39467)
I20260926 22:49:13.340632 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:33033
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:13.359601 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38259
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:13.437582 17807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:13.437803 17807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:13.437825 17807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:13.438971 17807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:13.439013 17807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:13.439037 17807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:13.440472 17807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:38259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:13.440676 17807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:13.440886 17807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:49:13.445497 17807 server_base.cc:1034] running on GCE node
W20260926 22:49:13.445434 17812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:13.445434 17815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:13.445443 17813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:13.445871 17807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:13.446125 17807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:13.447252 17807 hybrid_clock.cc:648] HybridClock initialized: now 1790462953447244 us; error 26 us; skew 500 ppm
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:13.453711 17807 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:13.454164 17807 webserver.cc:460] Webserver started at http://127.13.148.3:45313/ using document root <none> and password file <none>
I20260926 22:49:13.454360 17807 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:13.454419 17807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:13.454547 17807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:13.455580 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "5aabb4f968694150870f37de1de4f827"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "209fa282d7842df878840a385f98528c"
server_key_iv: "65809e6e64c62823dae625deb8ab9b45"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.455902 17807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "5aabb4f968694150870f37de1de4f827"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "209fa282d7842df878840a385f98528c"
server_key_iv: "65809e6e64c62823dae625deb8ab9b45"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.457150 17807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:13.457841 17822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.457978 17807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.458065 17807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "5aabb4f968694150870f37de1de4f827"
format_stamp: "Formatted at 2026-09-26 22:49:13 on dist-test-slave-98hb"
server_key: "209fa282d7842df878840a385f98528c"
server_key_iv: "65809e6e64c62823dae625deb8ab9b45"
server_key_version: "encryptionkey@0"
I20260926 22:49:13.458130 17807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:13.485639 17807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:13.487777 17807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:13.487910 17807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:13.488123 17807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:13.488421 17807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:13.488453 17807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.488476 17807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:13.488499 17807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:13.493803 17807 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:45975
I20260926 22:49:13.493860 17935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:45975 every 8 connection(s)
I20260926 22:49:13.494154 17807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:13.502547 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.494812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:55451 (local address 127.13.148.62:38259)
0926 22:49:13.494870 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:13.494873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:13.495071 (+ 198us) server_negotiation.cc:407] Connection header received
0926 22:49:13.495975 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:13.495976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:13.495986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:13.496034 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:13.496608 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.497145 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.498206 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.498395 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.498439 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.500371 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:13.500372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:13.500373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:13.500379 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:13.501553 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.501887 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.501887 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.501887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.501934 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.502103 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.502104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.502104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.502194 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:13.502236 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:13.502428 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 22:49:13.502482 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:49:13.502736 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.494802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:38259 (local address 127.13.148.3:55451)
0926 22:49:13.494961 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:13.494998 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:13.495895 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:13.496104 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:13.496107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:13.496357 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:13.496544 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:13.496549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.497219 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.497220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:13.498136 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:13.498139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.498452 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.498453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:13.498552 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.499012 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:13.499021 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:13.500292 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:13.501611 (+ 1319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:13.501613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:13.501615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:13.501841 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:13.501977 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:13.501978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:13.501978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:13.502066 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:13.502233 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:13.502236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:13.502346 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 22:49:13.502418 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 22:49:13.502529 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260926 22:49:13.503136 17936 heartbeater.cc:347] Connected to a master server at 127.13.148.62:38259
I20260926 22:49:13.503222 17936 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:13.503396 17936 heartbeater.cc:511] Master 127.13.148.62:38259 requested a full tablet report, sending...
I20260926 22:49:13.503722 17477 ts_manager.cc:194] Registered new tserver with Master: 5aabb4f968694150870f37de1de4f827 (127.13.148.3:45975)
I20260926 22:49:13.504061 17477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:55451
I20260926 22:49:13.504237 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 17807
I20260926 22:49:13.504354 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:13.504558 13904 external_mini_cluster.cc:1428] Setting key 0ab588a8fdae07d252ae201275b278a6
I20260926 22:49:13.505281 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:49:13.525475 17943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.518966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.13.148.1:41193)
0926 22:49:13.519121 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:13.519126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:13.519136 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:49:13.519159 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:13.519161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:13.519175 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:13.519235 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:13.519721 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.520403 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.521053 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.521347 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.521392 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.523275 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:13.523284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:13.523284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:13.523290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:13.524645 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.524894 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.524896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.524896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.524945 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.525057 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.525058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.525058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.525168 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:13.525282 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:13.525313 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 22:49:13.525370 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:13.532076 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:13.526567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.13.148.62:38259)
0926 22:49:13.526610 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:13.526611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:13.526621 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:49:13.526659 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:13.526659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:13.526667 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:13.526716 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:13.527078 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.527536 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.528131 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:13.528314 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:13.528362 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:13.530180 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:13.530182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:13.530182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:13.530189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:13.531332 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.531560 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.531561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.531561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.531603 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:13.531715 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:13.531715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:13.531715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:13.531835 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:13.531899 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:13.531960 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 22:49:13.532011 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 22:49:13.532279 17477 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35916
I20260926 22:49:13.532567 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 17539
I20260926 22:49:13.536533 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 17673
I20260926 22:49:13.540320 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 17807
I20260926 22:49:13.543850 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 17445
2026-09-26T22:49:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1729 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17948](info): set up 2 sockets
Sep 26 22:49:13 dist-test-slave-98hb krb5kdc[17948](info): commencing operation
krb5kdc: starting...
W20260926 22:49:14.493672 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:14 dist-test-slave-98hb krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:14.602691 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:33071
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:14.681457 17961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:14.681659 17961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:14.681680 17961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:14.682834 17961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:14.682875 17961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:14.682888 17961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:14.682899 17961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:14.682910 17961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:14.684373 17961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:14.684599 17961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:14.684806 17961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:14.689321 17969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:14.689357 17961 server_base.cc:1034] running on GCE node
W20260926 22:49:14.689380 17967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:14.689327 17966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:14.689718 17961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:14.690028 17961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:14.691157 17961 hybrid_clock.cc:648] HybridClock initialized: now 1790462954691146 us; error 26 us; skew 500 ppm
Sep 26 22:49:14 dist-test-slave-98hb krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:14.693441 17961 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:14.693709 17961 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:14.693805 17961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:14.693907 17961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:14.694869 17961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "e05f8c8233124d2eb09e84dc1b8543b2"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "7618e858c206febf9cf9d054823f407b"
server_key_iv: "2a289a9de1b76ffced47c905ce1376e0"
server_key_version: "encryptionkey@0"
I20260926 22:49:14.695147 17961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "e05f8c8233124d2eb09e84dc1b8543b2"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "7618e858c206febf9cf9d054823f407b"
server_key_iv: "2a289a9de1b76ffced47c905ce1376e0"
server_key_version: "encryptionkey@0"
I20260926 22:49:14.696266 17961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:49:14.696870 17975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:14.697007 17961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:14.697084 17961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "e05f8c8233124d2eb09e84dc1b8543b2"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "7618e858c206febf9cf9d054823f407b"
server_key_iv: "2a289a9de1b76ffced47c905ce1376e0"
server_key_version: "encryptionkey@0"
I20260926 22:49:14.697135 17961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:14.705060 17961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:14.707038 17961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:14.707142 17961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:14.710592 17961 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:33071
I20260926 22:49:14.710634 18027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:33071 every 8 connection(s)
I20260926 22:49:14.710875 17961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:14.711702 18028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:14.713822 18028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2: Bootstrap starting.
I20260926 22:49:14.714340 18028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:14.714560 18028 log.cc:824] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:14.715108 18028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2: No bootstrap required, opened a new log
I20260926 22:49:14.716459 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 17961
I20260926 22:49:14.716544 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:14.716696 13904 external_mini_cluster.cc:1428] Setting key 5c32c272e82cd495b6d3fa7ea8156a51
I20260926 22:49:14.716611 18028 raft_consensus.cc:348] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05f8c8233124d2eb09e84dc1b8543b2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33071 } }
I20260926 22:49:14.716734 18028 raft_consensus.cc:374] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:14.716765 18028 raft_consensus.cc:729] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05f8c8233124d2eb09e84dc1b8543b2, State: Initialized, Role: FOLLOWER
I20260926 22:49:14.716903 18028 consensus_queue.cc:260] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05f8c8233124d2eb09e84dc1b8543b2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33071 } }
I20260926 22:49:14.716984 18028 raft_consensus.cc:388] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:14.717015 18028 raft_consensus.cc:482] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:14.717060 18028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:14.718021 18028 raft_consensus.cc:504] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05f8c8233124d2eb09e84dc1b8543b2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33071 } }
I20260926 22:49:14.718115 18028 leader_election.cc:302] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05f8c8233124d2eb09e84dc1b8543b2; no voters:
I20260926 22:49:14.718268 18028 leader_election.cc:288] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:14.718331 18033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:14.718454 18033 raft_consensus.cc:686] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [term 1 LEADER]: Becoming Leader. State: Replica: e05f8c8233124d2eb09e84dc1b8543b2, State: Running, Role: LEADER
I20260926 22:49:14.718508 18028 sys_catalog.cc:564] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:14.718542 18033 consensus_queue.cc:237] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05f8c8233124d2eb09e84dc1b8543b2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33071 } }
I20260926 22:49:14.718864 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e05f8c8233124d2eb09e84dc1b8543b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05f8c8233124d2eb09e84dc1b8543b2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33071 } } }
I20260926 22:49:14.718956 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:14.718852 18033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e05f8c8233124d2eb09e84dc1b8543b2. Latest consensus state: current_term: 1 leader_uuid: "e05f8c8233124d2eb09e84dc1b8543b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05f8c8233124d2eb09e84dc1b8543b2" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 33071 } } }
I20260926 22:49:14.719123 18033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:14.719374 18040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:14.719996 18040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:14.721794 18040 catalog_manager.cc:1347] Generated new cluster ID: df6a42aaeddd4e8d8d086fbadba3a5af
I20260926 22:49:14.721840 18040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:49:14 dist-test-slave-98hb krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:14.727253 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:14.717078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.13.148.62:33071)
0926 22:49:14.717236 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:14.717245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:14.717358 (+ 113us) server_negotiation.cc:407] Connection header received
0926 22:49:14.717487 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:14.717495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:14.717564 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:14.717715 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:14.718309 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:14.719738 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:14.720733 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:14.721257 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:14.721354 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:14.723930 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:14.723933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:14.723934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:14.723944 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:14.725677 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:14.726117 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:14.726119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:14.726120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:14.726221 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:14.726409 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:14.726410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:14.726410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:14.726700 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:14.726830 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:14.726935 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 22:49:14.727059 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260926 22:49:14.731084 18040 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:14.731778 18040 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:14.736099 18040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2: Generated new TSK 0
I20260926 22:49:14.736238 18040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:14.761772 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33071
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:14.839686 18054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:14.839879 18054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:14.839910 18054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:14.841032 18054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:14.841070 18054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:14.841104 18054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:14.842623 18054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:14.842855 18054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:14.843055 18054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:14.847573 18060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:14.847594 18059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:14.847616 18062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:14.847606 18054 server_base.cc:1034] running on GCE node
I20260926 22:49:14.847956 18054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:14.848173 18054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:14.849325 18054 hybrid_clock.cc:648] HybridClock initialized: now 1790462954849313 us; error 24 us; skew 500 ppm
Sep 26 22:49:14 dist-test-slave-98hb krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:14.851630 18054 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:14.851951 18054 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:14.851998 18054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:14.852087 18054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:14.853039 18054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "df768268e7a343b3970881a5205d6fca"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "d23174926fe88365ece5ca5e5597f0c1"
server_key_iv: "28fecf46bb4c820ca257bfb4bfb1619f"
server_key_version: "encryptionkey@0"
I20260926 22:49:14.853322 18054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "df768268e7a343b3970881a5205d6fca"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "d23174926fe88365ece5ca5e5597f0c1"
server_key_iv: "28fecf46bb4c820ca257bfb4bfb1619f"
server_key_version: "encryptionkey@0"
I20260926 22:49:14.854555 18054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:14.855162 18068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:14.855302 18054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:14.855376 18054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "df768268e7a343b3970881a5205d6fca"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "d23174926fe88365ece5ca5e5597f0c1"
server_key_iv: "28fecf46bb4c820ca257bfb4bfb1619f"
server_key_version: "encryptionkey@0"
I20260926 22:49:14.855438 18054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:14.876139 18054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:14.878187 18054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:14.878326 18054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:14.878528 18054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:14.878813 18054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:14.878844 18054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:14.878866 18054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:14.878888 18054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:14.884227 18054 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:38947
I20260926 22:49:14.884274 18181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:38947 every 8 connection(s)
I20260926 22:49:14.884513 18054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:14.886574 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 18054
I20260926 22:49:14.886665 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:14.886821 13904 external_mini_cluster.cc:1428] Setting key f81b5eb845c2a94fc6cfe0747fbddaeb
Sep 26 22:49:14 dist-test-slave-98hb krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:14.893314 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:14.885141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42751 (local address 127.13.148.62:33071)
0926 22:49:14.885183 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:14.885185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:14.885396 (+ 211us) server_negotiation.cc:407] Connection header received
0926 22:49:14.885929 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:14.885930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:14.885947 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:14.885996 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:14.886972 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:14.887460 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:14.888415 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:14.888697 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:14.888765 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:14.890798 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:14.890799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:14.890799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:14.890807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:14.892029 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:14.892460 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:14.892460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:14.892461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:14.892507 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:14.892760 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:14.892761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:14.892761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:14.892853 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:14.892896 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:14.893186 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 22:49:14.893251 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:14.893561 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:14.885123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:33071 (local address 127.13.148.1:42751)
0926 22:49:14.885285 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:14.885320 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:14.885817 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:14.886075 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:14.886079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:14.886346 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:14.886893 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:14.886899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:14.887536 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:14.887537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:14.888333 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:14.888336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:14.888775 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:14.888776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:14.888887 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:14.889315 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:14.889323 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:14.890700 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:14.892104 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:14.892110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:14.892113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:14.892395 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:14.892558 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:14.892558 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:14.892559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:14.892689 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:14.892903 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:14.892908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:14.893066 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 22:49:14.893178 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 22:49:14.893335 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260926 22:49:14.894155 18182 heartbeater.cc:347] Connected to a master server at 127.13.148.62:33071
I20260926 22:49:14.894246 18182 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:14.894385 18182 heartbeater.cc:511] Master 127.13.148.62:33071 requested a full tablet report, sending...
I20260926 22:49:14.894860 17992 ts_manager.cc:194] Registered new tserver with Master: df768268e7a343b3970881a5205d6fca (127.13.148.1:38947)
I20260926 22:49:14.895460 17992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:42751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:14.910035 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33071
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:14.987493 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:14.987682 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:14.987703 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:14.988834 18187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:14.988876 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:14.988901 18187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:14.990388 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:14.990602 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:14.990809 18187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:14.995329 18195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:14.995335 18193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:14.995396 18192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:14.995568 18187 server_base.cc:1034] running on GCE node
I20260926 22:49:14.995777 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:14.995995 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:14.997128 18187 hybrid_clock.cc:648] HybridClock initialized: now 1790462954997118 us; error 33 us; skew 500 ppm
Sep 26 22:49:14 dist-test-slave-98hb krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:14.999495 18187 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:14.999830 18187 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:14.999881 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:15.000001 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:15.001021 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "44989be36c454d58b665718b7c8cb282"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "c7951e08b27fe3f80b2ea281f5edeeba"
server_key_iv: "826f0b931b72ac9172e16250028eb6b1"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.001334 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "44989be36c454d58b665718b7c8cb282"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "c7951e08b27fe3f80b2ea281f5edeeba"
server_key_iv: "826f0b931b72ac9172e16250028eb6b1"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.002527 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:15.003167 18201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.003311 18187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:15.003389 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "44989be36c454d58b665718b7c8cb282"
format_stamp: "Formatted at 2026-09-26 22:49:14 on dist-test-slave-98hb"
server_key: "c7951e08b27fe3f80b2ea281f5edeeba"
server_key_iv: "826f0b931b72ac9172e16250028eb6b1"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.003445 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:15.021221 18187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:15.023064 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:15.023197 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:15.023406 18187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:15.023685 18187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:15.023725 18187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.023752 18187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:15.023775 18187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.028903 18187 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:38783
I20260926 22:49:15.028940 18314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:38783 every 8 connection(s)
I20260926 22:49:15.029178 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:15.034401 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 18187
I20260926 22:49:15.034498 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:15.034657 13904 external_mini_cluster.cc:1428] Setting key edbf34229855c9d2210488abdfc7c490
Sep 26 22:49:15 dist-test-slave-98hb krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462954, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:15.037559 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.029813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:40537 (local address 127.13.148.62:33071)
0926 22:49:15.029858 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.029859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.030061 (+ 202us) server_negotiation.cc:407] Connection header received
0926 22:49:15.030546 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.030548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.030559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.030606 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.031497 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.031994 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.032945 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.033148 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.033211 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.035170 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.035171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.035172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:15.035179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.036381 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:15.036722 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:15.036723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:15.036724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:15.036774 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:15.036983 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:15.036984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:15.036984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:15.037087 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.037140 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.037388 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.037471 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:15.037709 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.029804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:33071 (local address 127.13.148.2:40537)
0926 22:49:15.029954 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.029988 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.030457 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.030686 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.030690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.030942 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.031408 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.031413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.032075 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.032077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.032866 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.032868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.033215 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.033215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.033336 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.033728 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:15.033760 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.035080 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.036453 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:15.036459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:15.036461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:15.036658 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:15.036826 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:15.036827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:15.036827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:15.036930 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:15.037139 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.037145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.037276 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.037360 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.037503 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260926 22:49:15.038095 18315 heartbeater.cc:347] Connected to a master server at 127.13.148.62:33071
I20260926 22:49:15.038182 18315 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:15.038331 18315 heartbeater.cc:511] Master 127.13.148.62:33071 requested a full tablet report, sending...
I20260926 22:49:15.038667 17992 ts_manager.cc:194] Registered new tserver with Master: 44989be36c454d58b665718b7c8cb282 (127.13.148.2:38783)
I20260926 22:49:15.039027 17992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:40537
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:15.056113 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33071
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:15.134680 18320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:15.134883 18320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:15.134915 18320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:15.136085 18320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:15.136125 18320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:15.136149 18320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:15.137566 18320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:15.137769 18320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:15.137964 18320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:15.142591 18325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:15.142554 18328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:15.142607 18326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:15.142618 18320 server_base.cc:1034] running on GCE node
I20260926 22:49:15.142958 18320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:15.143177 18320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:15.144335 18320 hybrid_clock.cc:648] HybridClock initialized: now 1790462955144323 us; error 26 us; skew 500 ppm
Sep 26 22:49:15 dist-test-slave-98hb krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462955, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:15.146733 18320 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:15.147045 18320 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:15.147096 18320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:15.147223 18320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:15.148211 18320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "ad1bda3316c44dab8f58e84314622908"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "967ade69736e22b7bd90b063d7ed0082"
server_key_iv: "0ca7226f3592d98d12345c35d723f2f9"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.148530 18320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "ad1bda3316c44dab8f58e84314622908"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "967ade69736e22b7bd90b063d7ed0082"
server_key_iv: "0ca7226f3592d98d12345c35d723f2f9"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.149676 18320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:15.150362 18334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.150503 18320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:15.150583 18320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "ad1bda3316c44dab8f58e84314622908"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "967ade69736e22b7bd90b063d7ed0082"
server_key_iv: "0ca7226f3592d98d12345c35d723f2f9"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.150645 18320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:15.167359 18320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:15.169126 18320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:15.169263 18320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:15.169467 18320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:15.169785 18320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:15.169821 18320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.169850 18320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:15.169873 18320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.175122 18320 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:35695
I20260926 22:49:15.175170 18447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:35695 every 8 connection(s)
I20260926 22:49:15.175405 18320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:15.180485 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 18320
I20260926 22:49:15.180591 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:15.180770 13904 external_mini_cluster.cc:1428] Setting key bc50f4435944089d97ba9a49fdc72aa8
Sep 26 22:49:15 dist-test-slave-98hb krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462955, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:15.183873 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.176031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:43863 (local address 127.13.148.62:33071)
0926 22:49:15.176147 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.176150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.176295 (+ 145us) server_negotiation.cc:407] Connection header received
0926 22:49:15.176730 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.176732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.176742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.176791 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.177646 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.178176 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.179132 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.179344 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.179406 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.181497 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.181498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.181498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:15.181506 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.182630 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:15.183003 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:15.183004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:15.183005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:15.183062 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:15.183277 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:15.183278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:15.183278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:15.183392 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.183447 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.183703 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.183785 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 22:49:15.184043 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.176019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:33071 (local address 127.13.148.3:43863)
0926 22:49:15.176182 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.176218 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.176655 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.176843 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.176846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.177089 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.177552 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.177557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.178240 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.178241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.179031 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.179034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.179416 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.179417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.179514 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.179918 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:15.179926 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.181409 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.182720 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:15.182726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:15.182728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:15.182934 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:15.183115 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:15.183116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:15.183116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:15.183223 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:15.183446 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.183452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.183586 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.183674 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.183835 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260926 22:49:15.184427 18448 heartbeater.cc:347] Connected to a master server at 127.13.148.62:33071
I20260926 22:49:15.184513 18448 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:15.184653 18448 heartbeater.cc:511] Master 127.13.148.62:33071 requested a full tablet report, sending...
I20260926 22:49:15.184963 17992 ts_manager.cc:194] Registered new tserver with Master: ad1bda3316c44dab8f58e84314622908 (127.13.148.3:35695)
I20260926 22:49:15.185329 17992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:43863
I20260926 22:49:15.192502 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:49:15.200165 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.193987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.13.148.62:33071)
0926 22:49:15.194390 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.194392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.194402 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:49:15.194440 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.194441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.194452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.194512 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.195206 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.195778 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.196655 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.196873 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.197001 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.198160 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.198161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.198162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:15.198169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.199159 (+ 990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:15.199451 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:15.199452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:15.199452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:15.199503 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:15.199658 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:15.199659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:15.199659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:15.199806 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.199895 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.199997 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.200076 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376}
I20260926 22:49:15.201695 17992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:15.203473 17992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:15.209482 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.205526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.13.148.2:38783)
0926 22:49:15.205695 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.205700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.205710 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:49:15.206055 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.206058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.206073 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.206285 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:15.206358 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.207422 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.208648 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.209122 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.209186 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.209256 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.209367 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.209403 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
I20260926 22:49:15.209482 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.205333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:38783 (local address 127.0.0.1:55304)
0926 22:49:15.205500 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.205538 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.205601 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.206163 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.206164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.206166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:15.206216 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.206218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.207520 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.207521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.208593 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.208594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.209196 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.209197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.209273 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.209285 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.209344 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.209395 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":28,"thread_start_us":61,"threads_started":1}
I20260926 22:49:15.209647 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.205567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.13.148.3:35695)
0926 22:49:15.205724 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.205727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.205967 (+ 240us) server_negotiation.cc:407] Connection header received
0926 22:49:15.206037 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.206040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.206056 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.206138 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:15.206313 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.207440 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.208449 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.209147 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.209215 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.209269 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.209525 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.209563 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260926 22:49:15.209642 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.205512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.13.148.1:38947)
0926 22:49:15.205715 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.205719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.205751 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:49:15.205826 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.205828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.205843 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.205913 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:15.206237 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.207042 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.208566 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.209236 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.209290 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.209334 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.209525 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.209563 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260926 22:49:15.210034 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.205360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:35695 (local address 127.0.0.1:51242)
0926 22:49:15.205901 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.205911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.205972 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.206216 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.206217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.206219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:15.206247 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.206248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.207531 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.207531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.208373 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.208375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.209304 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.209305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.209363 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.209367 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.209414 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.209454 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":34,"thread_start_us":68,"threads_started":1}
I20260926 22:49:15.210129 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.205335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:38947 (local address 127.0.0.1:46672)
0926 22:49:15.205668 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.205677 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.205758 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.206027 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.206030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.206033 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:15.206152 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.206154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.207124 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.207125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.208492 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.208495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.209360 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.209361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.209411 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.209414 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.209452 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.209480 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":60,"threads_started":1}
I20260926 22:49:15.210690 18249 tablet_service.cc:1467] Processing CreateTablet for tablet 70809578a38444c780b6c267a21ee7f3 (DEFAULT_TABLE table=test-table [id=aa685268af9a4d77a18487e0b207d744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:15.210983 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70809578a38444c780b6c267a21ee7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:15.213340 18462 tablet_bootstrap.cc:492] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282: Bootstrap starting.
I20260926 22:49:15.213716 18382 tablet_service.cc:1467] Processing CreateTablet for tablet 70809578a38444c780b6c267a21ee7f3 (DEFAULT_TABLE table=test-table [id=aa685268af9a4d77a18487e0b207d744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:15.213977 18462 tablet_bootstrap.cc:654] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:15.214011 18382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70809578a38444c780b6c267a21ee7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:15.214178 18116 tablet_service.cc:1467] Processing CreateTablet for tablet 70809578a38444c780b6c267a21ee7f3 (DEFAULT_TABLE table=test-table [id=aa685268af9a4d77a18487e0b207d744]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:15.214471 18116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70809578a38444c780b6c267a21ee7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:15.214522 18462 log.cc:824] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:15.215359 18462 tablet_bootstrap.cc:492] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282: No bootstrap required, opened a new log
I20260926 22:49:15.215416 18462 ts_tablet_manager.cc:1397] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:15.216339 18464 tablet_bootstrap.cc:492] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: Bootstrap starting.
I20260926 22:49:15.216964 18464 tablet_bootstrap.cc:654] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:15.217399 18464 log.cc:824] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:15.217303 18462 raft_consensus.cc:348] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.217473 18462 raft_consensus.cc:374] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:15.217509 18462 raft_consensus.cc:729] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44989be36c454d58b665718b7c8cb282, State: Initialized, Role: FOLLOWER
I20260926 22:49:15.217501 18465 tablet_bootstrap.cc:492] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca: Bootstrap starting.
I20260926 22:49:15.217630 18462 consensus_queue.cc:260] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.217924 18462 ts_tablet_manager.cc:1428] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:49:15.217955 18315 heartbeater.cc:503] Master 127.13.148.62:33071 was elected leader, sending a full tablet report...
I20260926 22:49:15.218101 18465 tablet_bootstrap.cc:654] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:15.218719 18465 log.cc:824] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:15.219466 18465 tablet_bootstrap.cc:492] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca: No bootstrap required, opened a new log
I20260926 22:49:15.219538 18465 ts_tablet_manager.cc:1397] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:15.219508 18464 tablet_bootstrap.cc:492] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: No bootstrap required, opened a new log
I20260926 22:49:15.219601 18464 ts_tablet_manager.cc:1397] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:49:15.221464 18464 raft_consensus.cc:348] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.221464 18465 raft_consensus.cc:348] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.221627 18464 raft_consensus.cc:374] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:15.221627 18465 raft_consensus.cc:374] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:15.221661 18465 raft_consensus.cc:729] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df768268e7a343b3970881a5205d6fca, State: Initialized, Role: FOLLOWER
I20260926 22:49:15.221661 18464 raft_consensus.cc:729] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1bda3316c44dab8f58e84314622908, State: Initialized, Role: FOLLOWER
I20260926 22:49:15.221802 18465 consensus_queue.cc:260] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.221827 18464 consensus_queue.cc:260] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.222074 18465 ts_tablet_manager.cc:1428] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:49:15.222074 18464 ts_tablet_manager.cc:1428] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:49:15.222090 18448 heartbeater.cc:503] Master 127.13.148.62:33071 was elected leader, sending a full tablet report...
I20260926 22:49:15.222318 18182 heartbeater.cc:503] Master 127.13.148.62:33071 was elected leader, sending a full tablet report...
W20260926 22:49:15.279702 18316 tablet.cc:2369] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:15.332633 18467 raft_consensus.cc:482] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:15.332763 18467 raft_consensus.cc:504] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.333115 18467 leader_election.cc:288] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df768268e7a343b3970881a5205d6fca (127.13.148.1:38947), ad1bda3316c44dab8f58e84314622908 (127.13.148.3:35695)
I20260926 22:49:15.335846 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.333280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:38947 (local address 127.13.148.2:59953)
0926 22:49:15.333335 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.333346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.333385 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.333534 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.333535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.333536 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:15.333607 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.333609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.334269 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.334270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.335015 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.335017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.335608 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.335608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.335683 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.335687 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.335751 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.335784 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:49:15.335870 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.333350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:59953 (local address 127.13.148.1:38947)
0926 22:49:15.333397 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.333399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.333406 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:15.333438 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.333439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.333448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.333521 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:15.333669 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.334200 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.335119 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.335537 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.335599 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.335606 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.335765 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.335803 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:49:15.335943 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.333280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:35695 (local address 127.13.148.2:49591)
0926 22:49:15.333469 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.333479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.333536 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.333650 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.333651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.333652 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:15.333722 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.333723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.334364 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.334364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.335175 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:15.335177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.335710 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.335711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:15.335782 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.335787 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.335839 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.335874 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":12,"thread_start_us":96,"threads_started":1}
I20260926 22:49:15.335944 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.333345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:49591 (local address 127.13.148.3:35695)
0926 22:49:15.333393 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.333395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.333545 (+ 150us) server_negotiation.cc:407] Connection header received
0926 22:49:15.333577 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.333578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.333589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.333643 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:15.333815 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.334272 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.335260 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.335637 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.335701 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.335709 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.335854 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.335887 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:49:15.336202 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70809578a38444c780b6c267a21ee7f3" candidate_uuid: "44989be36c454d58b665718b7c8cb282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df768268e7a343b3970881a5205d6fca" is_pre_election: true
I20260926 22:49:15.336287 18402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70809578a38444c780b6c267a21ee7f3" candidate_uuid: "44989be36c454d58b665718b7c8cb282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad1bda3316c44dab8f58e84314622908" is_pre_election: true
I20260926 22:49:15.336350 18136 raft_consensus.cc:2454] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44989be36c454d58b665718b7c8cb282 in term 0.
I20260926 22:49:15.336429 18402 raft_consensus.cc:2454] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44989be36c454d58b665718b7c8cb282 in term 0.
I20260926 22:49:15.336615 18205 leader_election.cc:302] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44989be36c454d58b665718b7c8cb282, df768268e7a343b3970881a5205d6fca; no voters:
I20260926 22:49:15.336753 18467 raft_consensus.cc:2781] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:15.336812 18467 raft_consensus.cc:482] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:15.336826 18467 raft_consensus.cc:3037] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:15.337486 18467 raft_consensus.cc:504] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.337595 18467 leader_election.cc:288] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [CANDIDATE]: Term 1 election: Requested vote from peers df768268e7a343b3970881a5205d6fca (127.13.148.1:38947), ad1bda3316c44dab8f58e84314622908 (127.13.148.3:35695)
I20260926 22:49:15.337783 18402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70809578a38444c780b6c267a21ee7f3" candidate_uuid: "44989be36c454d58b665718b7c8cb282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad1bda3316c44dab8f58e84314622908"
I20260926 22:49:15.337796 18136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70809578a38444c780b6c267a21ee7f3" candidate_uuid: "44989be36c454d58b665718b7c8cb282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df768268e7a343b3970881a5205d6fca"
I20260926 22:49:15.337852 18402 raft_consensus.cc:3037] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:15.337867 18136 raft_consensus.cc:3037] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:15.338464 18402 raft_consensus.cc:2454] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44989be36c454d58b665718b7c8cb282 in term 1.
I20260926 22:49:15.338585 18136 raft_consensus.cc:2454] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44989be36c454d58b665718b7c8cb282 in term 1.
I20260926 22:49:15.338620 18202 leader_election.cc:302] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44989be36c454d58b665718b7c8cb282, ad1bda3316c44dab8f58e84314622908; no voters:
I20260926 22:49:15.338688 18467 raft_consensus.cc:2781] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:15.338819 18467 raft_consensus.cc:686] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 1 LEADER]: Becoming Leader. State: Replica: 44989be36c454d58b665718b7c8cb282, State: Running, Role: LEADER
I20260926 22:49:15.338905 18467 consensus_queue.cc:237] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } }
I20260926 22:49:15.339707 17991 catalog_manager.cc:5495] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44989be36c454d58b665718b7c8cb282 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "44989be36c454d58b665718b7c8cb282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44989be36c454d58b665718b7c8cb282" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:15.351788 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.349564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.13.148.2:38783)
0926 22:49:15.349607 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.349609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.349617 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:15.349682 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.349683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.349709 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.349804 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:15.349925 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.350378 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.351091 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:15.351287 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:15.351339 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:15.351483 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:15.351575 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:15.351640 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.351674 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.351699 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:49:15.353428 18402 raft_consensus.cc:1261] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 1 FOLLOWER]: Refusing update from remote peer 44989be36c454d58b665718b7c8cb282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:15.353431 18136 raft_consensus.cc:1261] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [term 1 FOLLOWER]: Refusing update from remote peer 44989be36c454d58b665718b7c8cb282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:15.353709 18472 consensus_queue.cc:1035] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad1bda3316c44dab8f58e84314622908" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:15.353850 18467 consensus_queue.cc:1035] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df768268e7a343b3970881a5205d6fca" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:15.356062 18475 mvcc.cc:205] Tried to move back new op lower bound from 7333736265125236736 to 7333736265068646400. Current Snapshot: MvccSnapshot[applied={T|T < 7333736265125236736}]
I20260926 22:49:15.356395 18477 mvcc.cc:205] Tried to move back new op lower bound from 7333736265125236736 to 7333736265068646400. Current Snapshot: MvccSnapshot[applied={T|T < 7333736265125236736}]
I20260926 22:49:15.356694 18476 mvcc.cc:205] Tried to move back new op lower bound from 7333736265125236736 to 7333736265068646400. Current Snapshot: MvccSnapshot[applied={T|T < 7333736265125236736}]
I20260926 22:49:15.358654 17991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:15.358744 17991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:15.359437 17991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e05f8c8233124d2eb09e84dc1b8543b2: Sending DeleteTablet for 3 replicas of tablet 70809578a38444c780b6c267a21ee7f3
I20260926 22:49:15.359717 18249 tablet_service.cc:1514] Processing DeleteTablet for tablet 70809578a38444c780b6c267a21ee7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:55304
I20260926 22:49:15.359774 18116 tablet_service.cc:1514] Processing DeleteTablet for tablet 70809578a38444c780b6c267a21ee7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46672
I20260926 22:49:15.359951 18485 tablet_replica.cc:330] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca: stopping tablet replica
I20260926 22:49:15.359973 18382 tablet_service.cc:1514] Processing DeleteTablet for tablet 70809578a38444c780b6c267a21ee7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51242
I20260926 22:49:15.360078 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 18054
I20260926 22:49:15.360069 18485 raft_consensus.cc:2229] T 70809578a38444c780b6c267a21ee7f3 P df768268e7a343b3970881a5205d6fca [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 22:49:15.364652 17979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:38947 (error 108)
I20260926 22:49:15.364864 18484 tablet_replica.cc:330] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282: stopping tablet replica
I20260926 22:49:15.364909 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 18187
I20260926 22:49:15.364966 18484 raft_consensus.cc:2229] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:15.365083 18484 raft_consensus.cc:2258] T 70809578a38444c780b6c267a21ee7f3 P 44989be36c454d58b665718b7c8cb282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:15.365222 18486 tablet_replica.cc:330] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: stopping tablet replica
I20260926 22:49:15.365309 18486 raft_consensus.cc:2229] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:15.365391 18486 raft_consensus.cc:2258] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:15.365808 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.365674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:38947 (local address 127.0.0.1:46676)
0926 22:49:15.365725 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.365776 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:49:15.365821 18486 ts_tablet_manager.cc:1905] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 22:49:15.366434 17979 catalog_manager.cc:4584] TS df768268e7a343b3970881a5205d6fca (127.13.148.1:38947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70809578a38444c780b6c267a21ee7f3: Network error: Client connection negotiation failed: client connection to 127.13.148.1:38947: connect: Connection refused (error 111)
I20260926 22:49:15.367436 18486 ts_tablet_manager.cc:1918] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:15.367506 18486 log.cc:1196] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/70809578a38444c780b6c267a21ee7f3
I20260926 22:49:15.367770 18486 ts_tablet_manager.cc:1939] T 70809578a38444c780b6c267a21ee7f3 P ad1bda3316c44dab8f58e84314622908: Deleting consensus metadata
I20260926 22:49:15.368145 17976 catalog_manager.cc:4841] TS ad1bda3316c44dab8f58e84314622908 (127.13.148.3:35695): tablet 70809578a38444c780b6c267a21ee7f3 (table test-table [id=aa685268af9a4d77a18487e0b207d744]) successfully deleted
I20260926 22:49:15.370329 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 18320
I20260926 22:49:15.370371 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.370282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:38783 (local address 127.0.0.1:55332)
0926 22:49:15.370320 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.370346 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.2:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 22:49:15.374086 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 17961
2026-09-26T22:49:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1830 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T22:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:15Z Disabled control of system clock
I20260926 22:49:15.398769 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36577
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 22:49:15.476354 18491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:15.476553 18491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:15.476572 18491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:15.477725 18491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:15.477784 18491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:15.477795 18491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:15.477808 18491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:15.477833 18491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:15.479166 18491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:15.479401 18491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:15.479673 18491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:15.484153 18496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:15.484196 18499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:15.484167 18497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:15.484345 18491 server_base.cc:1034] running on GCE node
I20260926 22:49:15.484565 18491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:15.484825 18491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:15.485955 18491 hybrid_clock.cc:648] HybridClock initialized: now 1790462955485942 us; error 29 us; skew 500 ppm
I20260926 22:49:15.487134 18491 webserver.cc:460] Webserver started at http://127.13.148.62:35361/ using document root <none> and password file <none>
I20260926 22:49:15.487358 18491 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:15.487402 18491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:15.487526 18491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:15.488559 18491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "970606fe06a5433f921fd15aadf07afa"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "55751fb4fddfeb3fb778726de42a6510"
server_key_iv: "e9f29a0058828ea1ff529e6907bf2373"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.488938 18491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "970606fe06a5433f921fd15aadf07afa"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "55751fb4fddfeb3fb778726de42a6510"
server_key_iv: "e9f29a0058828ea1ff529e6907bf2373"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.490248 18491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:15.490893 18505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.491035 18491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:15.491112 18491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "970606fe06a5433f921fd15aadf07afa"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "55751fb4fddfeb3fb778726de42a6510"
server_key_iv: "e9f29a0058828ea1ff529e6907bf2373"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.491173 18491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:15.494379 18491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:15.494612 18491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:15.494724 18491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:15.498157 18491 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36577
I20260926 22:49:15.498185 18557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36577 every 8 connection(s)
I20260926 22:49:15.498486 18491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:15.499060 18558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:15.501603 18558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa: Bootstrap starting.
I20260926 22:49:15.502136 18558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:15.502388 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 18491
I20260926 22:49:15.502390 18558 log.cc:824] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:15.502470 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:15.502624 13904 external_mini_cluster.cc:1428] Setting key 7f5f359ed7f5c1159d525847ce004f3a
I20260926 22:49:15.503010 18558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa: No bootstrap required, opened a new log
I20260926 22:49:15.504868 18558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970606fe06a5433f921fd15aadf07afa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36577 } }
I20260926 22:49:15.504993 18558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:15.505017 18558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970606fe06a5433f921fd15aadf07afa, State: Initialized, Role: FOLLOWER
I20260926 22:49:15.505160 18558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970606fe06a5433f921fd15aadf07afa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36577 } }
I20260926 22:49:15.505259 18558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:15.505296 18558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:15.505338 18558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:15.505463 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.503057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.13.148.62:36577)
0926 22:49:15.503254 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.503261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.503282 (+ 21us) server_negotiation.cc:407] Connection header received
0926 22:49:15.504113 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.504120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.504157 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.504280 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.504919 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.504923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.504923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.504927 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.504969 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.504978 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.505041 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.505231 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.505288 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260926 22:49:15.506286 18558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970606fe06a5433f921fd15aadf07afa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36577 } }
I20260926 22:49:15.506369 18558 leader_election.cc:302] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 970606fe06a5433f921fd15aadf07afa; no voters:
I20260926 22:49:15.506475 18558 leader_election.cc:288] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:15.506510 18563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:15.506642 18563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [term 1 LEADER]: Becoming Leader. State: Replica: 970606fe06a5433f921fd15aadf07afa, State: Running, Role: LEADER
I20260926 22:49:15.506690 18558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:15.506721 18563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970606fe06a5433f921fd15aadf07afa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36577 } }
I20260926 22:49:15.507095 18564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "970606fe06a5433f921fd15aadf07afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970606fe06a5433f921fd15aadf07afa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36577 } } }
I20260926 22:49:15.507206 18564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:15.507351 18565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 970606fe06a5433f921fd15aadf07afa. Latest consensus state: current_term: 1 leader_uuid: "970606fe06a5433f921fd15aadf07afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970606fe06a5433f921fd15aadf07afa" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36577 } } }
I20260926 22:49:15.507413 18565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:15.507658 18572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:15.508037 18572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:15.509310 18572 catalog_manager.cc:1347] Generated new cluster ID: 2770766a05ba4c208089da877dbce6f7
I20260926 22:49:15.509359 18572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:15.543202 18572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:15.543378 18572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:15.551877 18572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa: Generated new TSK 0
I20260926 22:49:15.552073 18572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:49:15.553951 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36577
--builtin_ntp_servers=127.13.148.20:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 22:49:15.635995 18582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:15.636217 18582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:15.636238 18582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:15.637367 18582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:15.637403 18582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:15.637427 18582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:15.638808 18582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:15.639021 18582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:15.639297 18582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:15.643862 18590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:15.643889 18588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:15.643953 18582 server_base.cc:1034] running on GCE node
W20260926 22:49:15.643862 18587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:15.644237 18582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:15.644475 18582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:15.645603 18582 hybrid_clock.cc:648] HybridClock initialized: now 1790462955645589 us; error 27 us; skew 500 ppm
I20260926 22:49:15.646780 18582 webserver.cc:460] Webserver started at http://127.13.148.1:40935/ using document root <none> and password file <none>
I20260926 22:49:15.647033 18582 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:15.647084 18582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:15.647209 18582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:15.648190 18582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "7d1e0093c7ff4267a6fa693136cdd775"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "8e4c695799df7de689c178581188c007"
server_key_iv: "7ea23a4f8829465f2f4347baf6a6c5d6"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.648497 18582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "7d1e0093c7ff4267a6fa693136cdd775"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "8e4c695799df7de689c178581188c007"
server_key_iv: "7ea23a4f8829465f2f4347baf6a6c5d6"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.649642 18582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:15.650393 18596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.650570 18582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.650655 18582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "7d1e0093c7ff4267a6fa693136cdd775"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "8e4c695799df7de689c178581188c007"
server_key_iv: "7ea23a4f8829465f2f4347baf6a6c5d6"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.650730 18582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:15.654676 18582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:15.654906 18582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:15.655014 18582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:15.655182 18582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:15.655447 18582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:15.655481 18582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.655506 18582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:15.655529 18582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.660563 18582 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:45701
I20260926 22:49:15.660620 18709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:45701 every 8 connection(s)
I20260926 22:49:15.660881 18582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:15.663492 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.661490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:46493 (local address 127.13.148.62:36577)
0926 22:49:15.661538 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.661539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.661726 (+ 187us) server_negotiation.cc:407] Connection header received
0926 22:49:15.662194 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.662195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.662202 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.662258 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.663264 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.663265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.663266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.663270 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.663284 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.663289 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.663338 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.663409 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.663424 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:49:15.663658 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.661467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36577 (local address 127.13.148.1:46493)
0926 22:49:15.661626 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.661655 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.662103 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.662340 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.662344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.662352 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.663131 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:15.663132 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.663157 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:15.663182 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.663340 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.663345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.663348 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.663415 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.663480 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260926 22:49:15.664062 18710 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36577
I20260926 22:49:15.664130 18710 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:15.664263 18710 heartbeater.cc:511] Master 127.13.148.62:36577 requested a full tablet report, sending...
I20260926 22:49:15.664690 18522 ts_manager.cc:194] Registered new tserver with Master: 7d1e0093c7ff4267a6fa693136cdd775 (127.13.148.1:45701)
I20260926 22:49:15.668944 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 18582
I20260926 22:49:15.669039 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:15.669200 13904 external_mini_cluster.cc:1428] Setting key a466437db3f557cca3eb52723ba2ea2d
I20260926 22:49:15.669967 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36577
--builtin_ntp_servers=127.13.148.20:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 22:49:15.747200 18713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:15.747407 18713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:15.747435 18713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:15.748541 18713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:15.748577 18713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:15.748608 18713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:15.749940 18713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:15.750161 18713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:15.750419 18713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:15.755141 18718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:15.755184 18713 server_base.cc:1034] running on GCE node
W20260926 22:49:15.755147 18721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:15.755147 18719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:15.755546 18713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:15.755793 18713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:15.756925 18713 hybrid_clock.cc:648] HybridClock initialized: now 1790462955756911 us; error 27 us; skew 500 ppm
I20260926 22:49:15.758050 18713 webserver.cc:460] Webserver started at http://127.13.148.2:37531/ using document root <none> and password file <none>
I20260926 22:49:15.758255 18713 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:15.758301 18713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:15.758417 18713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:15.759426 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "e6a7df3d79f94935b8afee059b0d0b78"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "ff94c2c291c30299241058b4d1fae0a7"
server_key_iv: "ec7472c0739a550062b9a3ac60130f68"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.759743 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "e6a7df3d79f94935b8afee059b0d0b78"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "ff94c2c291c30299241058b4d1fae0a7"
server_key_iv: "ec7472c0739a550062b9a3ac60130f68"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.760944 18713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:15.761538 18727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.761685 18713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:15.761781 18713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "e6a7df3d79f94935b8afee059b0d0b78"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "ff94c2c291c30299241058b4d1fae0a7"
server_key_iv: "ec7472c0739a550062b9a3ac60130f68"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.761843 18713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:15.764973 18713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:15.765205 18713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:15.765316 18713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:15.765496 18713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:15.765813 18713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:15.765856 18713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.765882 18713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:15.765903 18713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.771111 18713 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:46555
I20260926 22:49:15.771147 18840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:46555 every 8 connection(s)
I20260926 22:49:15.771435 18713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:15.773859 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 18713
I20260926 22:49:15.773962 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:15.774163 13904 external_mini_cluster.cc:1428] Setting key d5bee8e8bbe928b30e3a729efbd0ca8d
I20260926 22:49:15.774220 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.772075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:40515 (local address 127.13.148.62:36577)
0926 22:49:15.772122 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.772124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.772352 (+ 228us) server_negotiation.cc:407] Connection header received
0926 22:49:15.772792 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.772793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.772798 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.772845 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.773993 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.773994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.773995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.773999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.774016 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.774021 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.774070 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.774152 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.774164 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:49:15.774472 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.772063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36577 (local address 127.13.148.2:40515)
0926 22:49:15.772240 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.772270 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.772715 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.772904 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.772907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.772914 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.773839 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:15.773840 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.773877 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:15.773912 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.774083 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.774089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.774092 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.774161 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.774223 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260926 22:49:15.774860 18841 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36577
I20260926 22:49:15.774940 18841 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:15.775074 18841 heartbeater.cc:511] Master 127.13.148.62:36577 requested a full tablet report, sending...
I20260926 22:49:15.775082 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36577
--builtin_ntp_servers=127.13.148.20:37839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 22:49:15.775383 18522 ts_manager.cc:194] Registered new tserver with Master: e6a7df3d79f94935b8afee059b0d0b78 (127.13.148.2:46555)
W20260926 22:49:15.854295 18844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:15.854483 18844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:15.854518 18844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:15.855700 18844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:15.855737 18844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:15.855770 18844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:15.857081 18844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:15.857306 18844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:15.857568 18844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:49:15.862110 18844 server_base.cc:1034] running on GCE node
W20260926 22:49:15.862078 18849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:15.862084 18850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:15.862076 18852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:15.862429 18844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:15.862659 18844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:15.863766 18844 hybrid_clock.cc:648] HybridClock initialized: now 1790462955863752 us; error 27 us; skew 500 ppm
I20260926 22:49:15.864944 18844 webserver.cc:460] Webserver started at http://127.13.148.3:35419/ using document root <none> and password file <none>
I20260926 22:49:15.865166 18844 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:15.865223 18844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:15.865341 18844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:15.866415 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "ba28367ad6ef468c9bb43b3f63c4d732"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "f94f9a771b3be48b6f0263fa8ef37c07"
server_key_iv: "f1bfa3da63d7dbe21678ebc0b28322d4"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.866724 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "ba28367ad6ef468c9bb43b3f63c4d732"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "f94f9a771b3be48b6f0263fa8ef37c07"
server_key_iv: "f1bfa3da63d7dbe21678ebc0b28322d4"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.867869 18844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:15.868490 18858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.868624 18844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:15.868696 18844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "ba28367ad6ef468c9bb43b3f63c4d732"
format_stamp: "Formatted at 2026-09-26 22:49:15 on dist-test-slave-98hb"
server_key: "f94f9a771b3be48b6f0263fa8ef37c07"
server_key_iv: "f1bfa3da63d7dbe21678ebc0b28322d4"
server_key_version: "encryptionkey@0"
I20260926 22:49:15.868762 18844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:15.872025 18844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:15.872255 18844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:15.872367 18844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:15.872550 18844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:15.872817 18844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:15.872848 18844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.872875 18844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:15.872897 18844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:15.878100 18844 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:34799
I20260926 22:49:15.878154 18971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:34799 every 8 connection(s)
I20260926 22:49:15.878415 18844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:15.879540 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 18844
I20260926 22:49:15.879624 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:15.879825 13904 external_mini_cluster.cc:1428] Setting key d365b05d3111cea1452849d0a4d9562d
I20260926 22:49:15.881073 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.879035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:53789 (local address 127.13.148.62:36577)
0926 22:49:15.879095 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.879097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.879289 (+ 192us) server_negotiation.cc:407] Connection header received
0926 22:49:15.879760 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.879761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.879768 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.879845 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.880875 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.880876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.880876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.880880 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.880893 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.880896 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.880934 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.880996 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.881009 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:49:15.881282 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.879009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36577 (local address 127.13.148.3:53789)
0926 22:49:15.879170 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.879208 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.879673 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.879910 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.879914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.879924 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.880753 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:15.880753 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.880779 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:15.880807 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.880935 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.880937 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.880954 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.881001 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.881045 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260926 22:49:15.881634 18972 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36577
I20260926 22:49:15.881707 18972 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:15.881867 18972 heartbeater.cc:511] Master 127.13.148.62:36577 requested a full tablet report, sending...
I20260926 22:49:15.882153 18522 ts_manager.cc:194] Registered new tserver with Master: ba28367ad6ef468c9bb43b3f63c4d732 (127.13.148.3:34799)
I20260926 22:49:15.891292 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:49:15.893345 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.892476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.13.148.62:36577)
0926 22:49:15.892544 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.892546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.892613 (+ 67us) server_negotiation.cc:407] Connection header received
0926 22:49:15.892654 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.892655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.892659 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.892702 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.893171 (+ 469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.893172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.893173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.893176 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.893187 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.893190 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.893224 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.893269 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.893279 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:49:15.894668 18522 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:15.896122 18522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:15.899535 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.897992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34799 (local address 127.0.0.1:42430)
0926 22:49:15.898301 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.898312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.898357 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.898643 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.898648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.898655 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.899128 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:15.899129 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.899144 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:15.899167 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.899383 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.899390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.899392 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.899456 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.899469 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":107,"threads_started":1}
I20260926 22:49:15.899670 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.898375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.13.148.3:34799)
0926 22:49:15.898510 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.898514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.898522 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:15.898544 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.898546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.898555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.898626 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.899239 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.899247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.899247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.899251 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.899284 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.899306 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.899359 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.899575 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.899591 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260926 22:49:15.899919 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.897998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46555 (local address 127.0.0.1:58508)
0926 22:49:15.898338 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.898345 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.898377 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.898656 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.898657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.898658 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.899359 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:15.899360 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.899365 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:15.899375 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.899804 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.899805 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.899805 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.899854 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.899866 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":97,"threads_started":1}
I20260926 22:49:15.899921 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.898129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.13.148.1:45701)
0926 22:49:15.898278 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.898283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.898303 (+ 20us) server_negotiation.cc:407] Connection header received
0926 22:49:15.898329 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.898332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.898341 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.898408 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.899339 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.899345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.899346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.899350 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.899379 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.899406 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.899464 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.899526 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.899540 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260926 22:49:15.900038 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.898220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.13.148.2:46555)
0926 22:49:15.898533 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.898537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.898545 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:15.898564 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.898566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.898572 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.899291 (+ 719us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.899652 (+ 361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.899656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.899656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.899659 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.899689 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.899703 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.899808 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.899876 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.899891 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":68,"threads_started":1}
I20260926 22:49:15.900298 18906 tablet_service.cc:1467] Processing CreateTablet for tablet 3d579f7d27d54525a7d8a767b02277ca (DEFAULT_TABLE table=test-table [id=fd6121f6509a43e1ad3fbf6c89435892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:15.900583 18906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d579f7d27d54525a7d8a767b02277ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:15.900565 18775 tablet_service.cc:1467] Processing CreateTablet for tablet 3d579f7d27d54525a7d8a767b02277ca (DEFAULT_TABLE table=test-table [id=fd6121f6509a43e1ad3fbf6c89435892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:15.900878 18775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d579f7d27d54525a7d8a767b02277ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:15.901867 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.897979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45701 (local address 127.0.0.1:42436)
0926 22:49:15.898162 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.898210 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.898265 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.898868 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.898869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.898872 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.899241 (+ 369us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:15.899242 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.899248 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:15.899260 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.899469 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.899470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.899471 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.899521 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.899533 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":47,"threads_started":1}
I20260926 22:49:15.902506 18644 tablet_service.cc:1467] Processing CreateTablet for tablet 3d579f7d27d54525a7d8a767b02277ca (DEFAULT_TABLE table=test-table [id=fd6121f6509a43e1ad3fbf6c89435892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:15.902817 18644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d579f7d27d54525a7d8a767b02277ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:15.903087 18986 tablet_bootstrap.cc:492] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: Bootstrap starting.
I20260926 22:49:15.903765 18987 tablet_bootstrap.cc:492] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: Bootstrap starting.
I20260926 22:49:15.903962 18986 tablet_bootstrap.cc:654] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:15.904254 18986 log.cc:824] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:15.904358 18987 tablet_bootstrap.cc:654] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:15.904595 18987 log.cc:824] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:15.904947 18986 tablet_bootstrap.cc:492] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: No bootstrap required, opened a new log
I20260926 22:49:15.904997 18986 ts_tablet_manager.cc:1397] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:15.905272 18987 tablet_bootstrap.cc:492] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: No bootstrap required, opened a new log
I20260926 22:49:15.905321 18987 ts_tablet_manager.cc:1397] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:15.905448 18990 tablet_bootstrap.cc:492] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775: Bootstrap starting.
I20260926 22:49:15.906018 18990 tablet_bootstrap.cc:654] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:15.906311 18990 log.cc:824] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:15.906908 18990 tablet_bootstrap.cc:492] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775: No bootstrap required, opened a new log
I20260926 22:49:15.906967 18990 ts_tablet_manager.cc:1397] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:15.907481 18986 raft_consensus.cc:348] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.907657 18986 raft_consensus.cc:374] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:15.907708 18986 raft_consensus.cc:729] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba28367ad6ef468c9bb43b3f63c4d732, State: Initialized, Role: FOLLOWER
I20260926 22:49:15.907824 18986 consensus_queue.cc:260] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.908077 18986 ts_tablet_manager.cc:1428] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:49:15.907989 18987 raft_consensus.cc:348] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.908094 18972 heartbeater.cc:503] Master 127.13.148.62:36577 was elected leader, sending a full tablet report...
I20260926 22:49:15.908124 18987 raft_consensus.cc:374] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:15.908154 18987 raft_consensus.cc:729] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a7df3d79f94935b8afee059b0d0b78, State: Initialized, Role: FOLLOWER
I20260926 22:49:15.908257 18987 consensus_queue.cc:260] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.908470 18987 ts_tablet_manager.cc:1428] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:49:15.908506 18841 heartbeater.cc:503] Master 127.13.148.62:36577 was elected leader, sending a full tablet report...
I20260926 22:49:15.909170 18990 raft_consensus.cc:348] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.909329 18990 raft_consensus.cc:374] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:15.909369 18990 raft_consensus.cc:729] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e0093c7ff4267a6fa693136cdd775, State: Initialized, Role: FOLLOWER
I20260926 22:49:15.909497 18990 consensus_queue.cc:260] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.909756 18990 ts_tablet_manager.cc:1428] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:49:15.909777 18710 heartbeater.cc:503] Master 127.13.148.62:36577 was elected leader, sending a full tablet report...
W20260926 22:49:15.911296 18711 tablet.cc:2369] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:15.964848 18992 raft_consensus.cc:482] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:15.964967 18992 raft_consensus.cc:504] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.965329 18992 leader_election.cc:288] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e0093c7ff4267a6fa693136cdd775 (127.13.148.1:45701), e6a7df3d79f94935b8afee059b0d0b78 (127.13.148.2:46555)
I20260926 22:49:15.966508 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.965495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45701 (local address 127.13.148.3:60235)
0926 22:49:15.965559 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.965569 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.965605 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.965799 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.965799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.965802 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.966202 (+ 400us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:15.966202 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.966209 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:15.966220 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.966388 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.966389 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.966389 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.966434 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.966446 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 22:49:15.966521 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.965560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:60235 (local address 127.13.148.1:45701)
0926 22:49:15.965623 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.965625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.965651 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:49:15.965678 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.965679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.965684 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.965788 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.966302 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.966303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.966304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.966308 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.966326 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.966331 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.966381 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.966435 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.966448 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:49:15.966611 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.965570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:40995 (local address 127.13.148.2:46555)
0926 22:49:15.965617 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.965619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.965885 (+ 266us) server_negotiation.cc:407] Connection header received
0926 22:49:15.965924 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.965925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.965930 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.965983 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.966422 (+ 439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.966423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.966424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.966427 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.966443 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.966447 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.966497 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.966542 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.966553 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:49:15.966622 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.965495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46555 (local address 127.13.148.3:40995)
0926 22:49:15.965817 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:15.965826 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:15.965876 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:15.965991 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:15.965991 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:15.965993 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:15.966341 (+ 348us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:15.966341 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:15.966346 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:15.966353 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:15.966499 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:15.966499 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:15.966500 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:49:15.966541 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 22:49:15.966551 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":136,"thread_start_us":135,"threads_started":1}
I20260926 22:49:15.966805 18664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d579f7d27d54525a7d8a767b02277ca" candidate_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e0093c7ff4267a6fa693136cdd775" is_pre_election: true
I20260926 22:49:15.966953 18664 raft_consensus.cc:2454] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba28367ad6ef468c9bb43b3f63c4d732 in term 0.
I20260926 22:49:15.966922 18795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d579f7d27d54525a7d8a767b02277ca" candidate_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a7df3d79f94935b8afee059b0d0b78" is_pre_election: true
I20260926 22:49:15.967051 18795 raft_consensus.cc:2454] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba28367ad6ef468c9bb43b3f63c4d732 in term 0.
I20260926 22:49:15.967168 18859 leader_election.cc:302] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d1e0093c7ff4267a6fa693136cdd775, ba28367ad6ef468c9bb43b3f63c4d732; no voters:
I20260926 22:49:15.967284 18992 raft_consensus.cc:2781] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:15.967347 18992 raft_consensus.cc:482] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:15.967376 18992 raft_consensus.cc:3037] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:15.968040 18992 raft_consensus.cc:504] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.968158 18992 leader_election.cc:288] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1e0093c7ff4267a6fa693136cdd775 (127.13.148.1:45701), e6a7df3d79f94935b8afee059b0d0b78 (127.13.148.2:46555)
I20260926 22:49:15.968309 18795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d579f7d27d54525a7d8a767b02277ca" candidate_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a7df3d79f94935b8afee059b0d0b78"
I20260926 22:49:15.968382 18795 raft_consensus.cc:3037] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:15.969045 18795 raft_consensus.cc:2454] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba28367ad6ef468c9bb43b3f63c4d732 in term 1.
I20260926 22:49:15.969142 18664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d579f7d27d54525a7d8a767b02277ca" candidate_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e0093c7ff4267a6fa693136cdd775"
I20260926 22:49:15.969202 18861 leader_election.cc:302] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba28367ad6ef468c9bb43b3f63c4d732, e6a7df3d79f94935b8afee059b0d0b78; no voters:
I20260926 22:49:15.969223 18664 raft_consensus.cc:3037] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:15.969303 18992 raft_consensus.cc:2781] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:15.969448 18992 raft_consensus.cc:686] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 1 LEADER]: Becoming Leader. State: Replica: ba28367ad6ef468c9bb43b3f63c4d732, State: Running, Role: LEADER
I20260926 22:49:15.969537 18992 consensus_queue.cc:237] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } }
I20260926 22:49:15.970094 18664 raft_consensus.cc:2454] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba28367ad6ef468c9bb43b3f63c4d732 in term 1.
I20260926 22:49:15.970585 18521 catalog_manager.cc:5495] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba28367ad6ef468c9bb43b3f63c4d732 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba28367ad6ef468c9bb43b3f63c4d732" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:15.990871 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:15.990060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.13.148.3:34799)
0926 22:49:15.990109 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:15.990111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:15.990118 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:15.990183 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:15.990184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:15.990199 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:15.990248 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:15.990685 (+ 437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:15.990687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:15.990687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:15.990703 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:15.990717 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:15.990721 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:15.990761 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:15.990790 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 22:49:15.990802 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:49:15.992560 18664 raft_consensus.cc:1261] T 3d579f7d27d54525a7d8a767b02277ca P 7d1e0093c7ff4267a6fa693136cdd775 [term 1 FOLLOWER]: Refusing update from remote peer ba28367ad6ef468c9bb43b3f63c4d732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:15.992604 18795 raft_consensus.cc:1261] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 1 FOLLOWER]: Refusing update from remote peer ba28367ad6ef468c9bb43b3f63c4d732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:15.992908 18996 consensus_queue.cc:1035] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e0093c7ff4267a6fa693136cdd775" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:15.992997 18992 consensus_queue.cc:1035] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6a7df3d79f94935b8afee059b0d0b78" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:15.995358 18999 mvcc.cc:205] Tried to move back new op lower bound from 7333736267743137792 to 7333736267651837952. Current Snapshot: MvccSnapshot[applied={T|T < 7333736267743137792}]
I20260926 22:49:15.995735 19001 mvcc.cc:205] Tried to move back new op lower bound from 7333736267743137792 to 7333736267651837952. Current Snapshot: MvccSnapshot[applied={T|T < 7333736267743137792}]
I20260926 22:49:15.995820 19000 mvcc.cc:205] Tried to move back new op lower bound from 7333736267743137792 to 7333736267651837952. Current Snapshot: MvccSnapshot[applied={T|T < 7333736267743137792}]
I20260926 22:49:15.997447 18522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:15.997525 18522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:15.998220 18522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 970606fe06a5433f921fd15aadf07afa: Sending DeleteTablet for 3 replicas of tablet 3d579f7d27d54525a7d8a767b02277ca
I20260926 22:49:15.998525 18775 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d579f7d27d54525a7d8a767b02277ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:15 UTC) from {username='slave'} at 127.0.0.1:58508
I20260926 22:49:15.998535 18906 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d579f7d27d54525a7d8a767b02277ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:15 UTC) from {username='slave'} at 127.0.0.1:42430
I20260926 22:49:15.998750 19009 tablet_replica.cc:330] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: stopping tablet replica
I20260926 22:49:15.998808 18644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d579f7d27d54525a7d8a767b02277ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:15 UTC) from {username='slave'} at 127.0.0.1:42436
I20260926 22:49:15.998840 19009 raft_consensus.cc:2229] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:15.998927 19009 raft_consensus.cc:2258] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:15.998957 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 18582
I20260926 22:49:15.999945 19008 tablet_replica.cc:330] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: stopping tablet replica
I20260926 22:49:16.000026 19008 raft_consensus.cc:2229] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:16.000108 19008 raft_consensus.cc:2258] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:16.000385 19009 ts_tablet_manager.cc:1905] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:16.002127 19009 ts_tablet_manager.cc:1918] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:16.002188 19009 log.cc:1196] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/3d579f7d27d54525a7d8a767b02277ca
I20260926 22:49:16.002432 19009 ts_tablet_manager.cc:1939] T 3d579f7d27d54525a7d8a767b02277ca P ba28367ad6ef468c9bb43b3f63c4d732: Deleting consensus metadata
I20260926 22:49:16.002838 19008 ts_tablet_manager.cc:1905] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:16.002992 18507 catalog_manager.cc:4841] TS ba28367ad6ef468c9bb43b3f63c4d732 (127.13.148.3:34799): tablet 3d579f7d27d54525a7d8a767b02277ca (table test-table [id=fd6121f6509a43e1ad3fbf6c89435892]) successfully deleted
I20260926 22:49:16.004216 19008 ts_tablet_manager.cc:1918] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:16.004285 19008 log.cc:1196] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/3d579f7d27d54525a7d8a767b02277ca
I20260926 22:49:16.004500 19008 ts_tablet_manager.cc:1939] T 3d579f7d27d54525a7d8a767b02277ca P e6a7df3d79f94935b8afee059b0d0b78: Deleting consensus metadata
I20260926 22:49:16.004907 18508 catalog_manager.cc:4841] TS e6a7df3d79f94935b8afee059b0d0b78 (127.13.148.2:46555): tablet 3d579f7d27d54525a7d8a767b02277ca (table test-table [id=fd6121f6509a43e1ad3fbf6c89435892]) successfully deleted
W20260926 22:49:16.005892 18506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:45701 (error 108)
I20260926 22:49:16.006367 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.006246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45701 (local address 127.0.0.1:42446)
0926 22:49:16.006295 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:16.006334 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 22:49:16.006422 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 18713
W20260926 22:49:16.009079 18506 catalog_manager.cc:4584] TS 7d1e0093c7ff4267a6fa693136cdd775 (127.13.148.1:45701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d579f7d27d54525a7d8a767b02277ca: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45701: connect: Connection refused (error 111)
I20260926 22:49:16.011694 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 18844
I20260926 22:49:16.016012 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 18491
2026-09-26T22:49:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (641 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 22:49:16.033664 13904 mini_oidc.cc:152] Starting JWKS server
I20260926 22:49:16.034739 13904 webserver.cc:460] Webserver started at https://127.0.0.1:34705/ using document root <none> and password file <none>
I20260926 22:49:16.034817 13904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 22:49:16.034938 13904 webserver.cc:460] Webserver started at http://127.0.0.1:44323/ using document root <none> and password file <none>
2026-09-26T22:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:16Z Disabled control of system clock
I20260926 22:49:16.041968 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34657
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:34657
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34705/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/testchainca.cert with env {}
W20260926 22:49:16.118643 19017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:16.118861 19017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:16.118882 19017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:16.120031 19017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:16.120069 19017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:16.120080 19017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:16.120095 19017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:16.120117 19017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 22:49:16.120137 19017 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 22:49:16.120157 19017 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34705/jwks.json
I20260926 22:49:16.121682 19017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:34657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34705/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:16.121932 19017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:16.122128 19017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:16.126456 19022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:16.126492 19023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:16.126446 19025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:16.126662 19017 server_base.cc:1034] running on GCE node
I20260926 22:49:16.126884 19017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:16.127148 19017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:16.128281 19017 hybrid_clock.cc:648] HybridClock initialized: now 1790462956128263 us; error 28 us; skew 500 ppm
I20260926 22:49:16.129364 19017 webserver.cc:460] Webserver started at http://127.13.148.62:37539/ using document root <none> and password file <none>
I20260926 22:49:16.129587 19017 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:16.129634 19017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:16.129801 19017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:16.130779 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "81c1bc6c4e654b289d84e282ed11ddd7"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "5646043c180af41be957733ca4bcd4d8"
server_key_iv: "7db0373bb7f69a161b3039fe2de3da43"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.131088 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "81c1bc6c4e654b289d84e282ed11ddd7"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "5646043c180af41be957733ca4bcd4d8"
server_key_iv: "7db0373bb7f69a161b3039fe2de3da43"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.132196 19017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:16.132795 19031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:16.132942 19017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:16.133019 19017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "81c1bc6c4e654b289d84e282ed11ddd7"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "5646043c180af41be957733ca4bcd4d8"
server_key_iv: "7db0373bb7f69a161b3039fe2de3da43"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.133081 19017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:16.148401 19017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:16.148682 19017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:16.148818 19017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:16.152371 19017 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:34657
I20260926 22:49:16.152415 19083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:34657 every 8 connection(s)
I20260926 22:49:16.152711 19017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:16.153570 19084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:16.155557 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19017
I20260926 22:49:16.155660 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:16.155807 13904 external_mini_cluster.cc:1428] Setting key 7c6c2e163220de31c37d59168e96fef2
I20260926 22:49:16.155821 19084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7: Bootstrap starting.
I20260926 22:49:16.156481 19084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:16.156802 19084 log.cc:824] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:16.157490 19084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7: No bootstrap required, opened a new log
I20260926 22:49:16.159598 19084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c1bc6c4e654b289d84e282ed11ddd7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34657 } }
I20260926 22:49:16.159768 19084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:16.159808 19084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81c1bc6c4e654b289d84e282ed11ddd7, State: Initialized, Role: FOLLOWER
I20260926 22:49:16.159925 19084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c1bc6c4e654b289d84e282ed11ddd7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34657 } }
I20260926 22:49:16.159991 19084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:16.160025 19084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:16.160068 19084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:16.160717 19084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c1bc6c4e654b289d84e282ed11ddd7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34657 } }
I20260926 22:49:16.160818 19084 leader_election.cc:302] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81c1bc6c4e654b289d84e282ed11ddd7; no voters:
I20260926 22:49:16.160964 19084 leader_election.cc:288] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:16.160989 19089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:16.161132 19089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [term 1 LEADER]: Becoming Leader. State: Replica: 81c1bc6c4e654b289d84e282ed11ddd7, State: Running, Role: LEADER
I20260926 22:49:16.161216 19089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c1bc6c4e654b289d84e282ed11ddd7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34657 } }
I20260926 22:49:16.161226 19084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:16.161513 19091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81c1bc6c4e654b289d84e282ed11ddd7. Latest consensus state: current_term: 1 leader_uuid: "81c1bc6c4e654b289d84e282ed11ddd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c1bc6c4e654b289d84e282ed11ddd7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34657 } } }
I20260926 22:49:16.161600 19091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:16.161651 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.156243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.13.148.62:34657)
0926 22:49:16.156442 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:16.156451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:16.156484 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:49:16.156997 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:16.157005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:16.157077 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:16.157208 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:16.157374 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.158730 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.159468 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.160044 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.160160 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.160714 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:16.160719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:16.160719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:16.160913 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:16.160956 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:16.160970 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:16.161310 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:16.161343 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 22:49:16.161421 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260926 22:49:16.161824 19093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:16.162436 19093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:16.162465 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:49:16.163034 19090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81c1bc6c4e654b289d84e282ed11ddd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81c1bc6c4e654b289d84e282ed11ddd7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34657 } } }
I20260926 22:49:16.163110 19090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:16.166128 19093 catalog_manager.cc:1347] Generated new cluster ID: b3cbcdb99c1142e4a55b72cb09e4e94f
I20260926 22:49:16.166184 19093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:16.202206 19093 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:16.202921 19093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:16.207854 19093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81c1bc6c4e654b289d84e282ed11ddd7: Generated new TSK 0
I20260926 22:49:16.208037 19093 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 22:49:16.240777 19114 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 22:49:16.240986 19114 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 22:49:16.240993 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.218944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.13.148.62:34657)
0926 22:49:16.218984 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:16.218986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:16.219098 (+ 112us) server_negotiation.cc:407] Connection header received
0926 22:49:16.219201 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:16.219202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:16.219212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:16.219259 (+ 47us) server_negotiation.cc:226] Negotiated authn=JWT
0926 22:49:16.219389 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.219872 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.220930 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.221106 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.221148 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.221359 (+ 211us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 22:49:16.240766 (+ 19407us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 22:49:16.240929 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 22:49:16.241034 19087 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33154: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 22:49:16.241421 19113 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:16.218862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:34657 (local address 127.0.0.1:33154)
0926 22:49:16.219027 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:16.219037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:16.219106 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:16.219269 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:16.219270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:16.219272 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 22:49:16.219336 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:16.219338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.219952 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.219953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:16.220859 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:16.220862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.221170 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.221171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:16.221245 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.221269 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 22:49:16.240927 (+ 19658us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 22:49:16.241075 (+ 148us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.62:34657: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.148.62:34657: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 22:49:16.241878 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19017
2026-09-26T22:49:16Z chronyd exiting
I20260926 22:49:16.277222 13904 test_util.cc:175] -----------------------------------------------
I20260926 22:49:16.277298 13904 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790462935413609-13904-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (244 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 22:49:16.278033 13904 mini_oidc.cc:152] Starting JWKS server
I20260926 22:49:16.278680 13904 webserver.cc:460] Webserver started at https://127.0.0.1:35513/ using document root <none> and password file <none>
I20260926 22:49:16.278721 13904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 22:49:16.278829 13904 webserver.cc:460] Webserver started at http://127.0.0.1:37569/ using document root <none> and password file <none>
2026-09-26T22:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:16Z Disabled control of system clock
I20260926 22:49:16.285552 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36539
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36539
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35513/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/testchainca.cert with env {}
W20260926 22:49:16.362077 19120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:16.362293 19120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:16.362313 19120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:16.363405 19120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:16.363440 19120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:16.363451 19120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:16.363461 19120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:16.363471 19120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 22:49:16.363481 19120 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 22:49:16.363492 19120 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35513/jwks.json
I20260926 22:49:16.364907 19120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35513/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:16.365108 19120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:16.365275 19120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:16.369652 19126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:16.369652 19125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:16.369722 19120 server_base.cc:1034] running on GCE node
W20260926 22:49:16.369642 19128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:16.370076 19120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:16.370348 19120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:16.371490 19120 hybrid_clock.cc:648] HybridClock initialized: now 1790462956371477 us; error 26 us; skew 500 ppm
I20260926 22:49:16.372591 19120 webserver.cc:460] Webserver started at http://127.13.148.62:34863/ using document root <none> and password file <none>
I20260926 22:49:16.372790 19120 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:16.372838 19120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:16.372953 19120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:16.373999 19120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "b414d96342904fbb97b1f7c09349ba7d"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "d4a92b4e18987bf88fa250d2115f23b4"
server_key_iv: "c77c6c0f8b1043c62974e7696b63c1b7"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.374323 19120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "b414d96342904fbb97b1f7c09349ba7d"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "d4a92b4e18987bf88fa250d2115f23b4"
server_key_iv: "c77c6c0f8b1043c62974e7696b63c1b7"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.375434 19120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:16.376036 19134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:16.376168 19120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:16.376255 19120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "b414d96342904fbb97b1f7c09349ba7d"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "d4a92b4e18987bf88fa250d2115f23b4"
server_key_iv: "c77c6c0f8b1043c62974e7696b63c1b7"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.376312 19120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:16.388844 19120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:16.389096 19120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:16.389209 19120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:16.392686 19120 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36539
I20260926 22:49:16.392726 19186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36539 every 8 connection(s)
I20260926 22:49:16.393003 19120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:16.393605 19187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:16.396281 19187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d: Bootstrap starting.
I20260926 22:49:16.396806 19187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:16.397048 19187 log.cc:824] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:16.397639 19187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d: No bootstrap required, opened a new log
I20260926 22:49:16.398900 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19120
I20260926 22:49:16.399003 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:16.399184 13904 external_mini_cluster.cc:1428] Setting key fe83016432b251d2a5887af83b75099e
I20260926 22:49:16.399408 19187 raft_consensus.cc:348] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414d96342904fbb97b1f7c09349ba7d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36539 } }
I20260926 22:49:16.399565 19187 raft_consensus.cc:374] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:16.399601 19187 raft_consensus.cc:729] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b414d96342904fbb97b1f7c09349ba7d, State: Initialized, Role: FOLLOWER
I20260926 22:49:16.399709 19187 consensus_queue.cc:260] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414d96342904fbb97b1f7c09349ba7d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36539 } }
I20260926 22:49:16.399780 19187 raft_consensus.cc:388] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:16.399816 19187 raft_consensus.cc:482] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:16.399868 19187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:16.400497 19187 raft_consensus.cc:504] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414d96342904fbb97b1f7c09349ba7d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36539 } }
I20260926 22:49:16.400597 19187 leader_election.cc:302] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b414d96342904fbb97b1f7c09349ba7d; no voters:
I20260926 22:49:16.400739 19187 leader_election.cc:288] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:16.400962 19192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:16.400969 19187 sys_catalog.cc:564] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:16.401091 19192 raft_consensus.cc:686] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [term 1 LEADER]: Becoming Leader. State: Replica: b414d96342904fbb97b1f7c09349ba7d, State: Running, Role: LEADER
I20260926 22:49:16.401172 19192 consensus_queue.cc:237] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414d96342904fbb97b1f7c09349ba7d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36539 } }
I20260926 22:49:16.401487 19193 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b414d96342904fbb97b1f7c09349ba7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414d96342904fbb97b1f7c09349ba7d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36539 } } }
I20260926 22:49:16.401495 19194 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b414d96342904fbb97b1f7c09349ba7d. Latest consensus state: current_term: 1 leader_uuid: "b414d96342904fbb97b1f7c09349ba7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b414d96342904fbb97b1f7c09349ba7d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36539 } } }
I20260926 22:49:16.401566 19193 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:16.401571 19194 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:16.401811 19201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:16.402302 19201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:16.404695 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.399639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.13.148.62:36539)
0926 22:49:16.399911 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:16.399919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:16.399951 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:49:16.400073 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:16.400080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:16.400164 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:16.400318 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:16.400460 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.401839 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.402893 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.403461 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.403580 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.404127 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:16.404131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:16.404131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:16.404296 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:16.404330 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:16.404341 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:16.404395 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:16.404463 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 22:49:16.404536 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
I20260926 22:49:16.405547 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:49:16.406808 19201 catalog_manager.cc:1347] Generated new cluster ID: 25fb8beb1e5c4f898af967511f040ebf
I20260926 22:49:16.406862 19201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:16.419793 19201 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:16.420289 19201 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:16.423043 19201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b414d96342904fbb97b1f7c09349ba7d: Generated new TSK 0
I20260926 22:49:16.423220 19201 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 22:49:16.453685 19217 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 22:49:16.453795 19217 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 22:49:16.454056 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.432273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.13.148.62:36539)
0926 22:49:16.432324 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:16.432326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:16.432414 (+ 88us) server_negotiation.cc:407] Connection header received
0926 22:49:16.432498 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:16.432499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:16.432509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:16.432564 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0926 22:49:16.432702 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.433189 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.434327 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.434519 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.434564 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.434724 (+ 160us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 22:49:16.453863 (+ 19139us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 22:49:16.454003 (+ 140us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48336: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 22:49:16.454103 19191 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48336: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 22:49:16.454209 19216 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:16.432217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36539 (local address 127.0.0.1:48336)
0926 22:49:16.432353 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:16.432364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:16.432434 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:16.432583 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:16.432584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:16.432586 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 22:49:16.432645 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:16.432647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.433261 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.433262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:16.434257 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:16.434260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.434583 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.434584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:16.434643 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.434648 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 22:49:16.453977 (+ 19329us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 22:49:16.454096 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.62:36539: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35513/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260926 22:49:16.454493 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19120
2026-09-26T22:49:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 22:49:16.491250 13904 mini_oidc.cc:152] Starting JWKS server
I20260926 22:49:16.491897 13904 webserver.cc:460] Webserver started at https://127.0.0.1:35031/ using document root <none> and password file <none>
I20260926 22:49:16.491940 13904 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 22:49:16.492065 13904 webserver.cc:460] Webserver started at http://127.0.0.1:46661/ using document root <none> and password file <none>
2026-09-26T22:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:16Z Disabled control of system clock
I20260926 22:49:16.498844 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44791
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:46199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:44791
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35031/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 22:49:16.575477 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:16.575672 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:16.575691 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:16.576844 19223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:16.576884 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:16.576894 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:16.576916 19223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:16.576941 19223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 22:49:16.576961 19223 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 22:49:16.576982 19223 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35031/jwks.json
I20260926 22:49:16.578336 19223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:44791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35031/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:16.578553 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:16.578804 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:16.583119 19231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:16.583178 19223 server_base.cc:1034] running on GCE node
W20260926 22:49:16.583140 19228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:16.583144 19229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:16.583630 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:16.583895 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:16.585039 19223 hybrid_clock.cc:648] HybridClock initialized: now 1790462956585023 us; error 27 us; skew 500 ppm
I20260926 22:49:16.586169 19223 webserver.cc:460] Webserver started at http://127.13.148.62:44617/ using document root <none> and password file <none>
I20260926 22:49:16.586370 19223 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:16.586417 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:16.586540 19223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:16.587531 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "daceeb1ec01a41708d2cd151ad8eb75e"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "d64b624703c2b12263b04833aa2517e7"
server_key_iv: "56bae92bb583f93d21948b5ae5c76b2e"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.587857 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "daceeb1ec01a41708d2cd151ad8eb75e"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "d64b624703c2b12263b04833aa2517e7"
server_key_iv: "56bae92bb583f93d21948b5ae5c76b2e"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.589025 19223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:49:16.589617 19237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:16.589772 19223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:16.589852 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "daceeb1ec01a41708d2cd151ad8eb75e"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "d64b624703c2b12263b04833aa2517e7"
server_key_iv: "56bae92bb583f93d21948b5ae5c76b2e"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.589916 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:16.601994 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:16.602252 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:16.602371 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:16.605808 19223 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:44791
I20260926 22:49:16.605844 19289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:44791 every 8 connection(s)
I20260926 22:49:16.606137 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:16.606696 19290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:16.609223 19290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e: Bootstrap starting.
I20260926 22:49:16.609706 19290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:16.609970 19290 log.cc:824] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:16.610543 19290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e: No bootstrap required, opened a new log
I20260926 22:49:16.612003 19290 raft_consensus.cc:348] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daceeb1ec01a41708d2cd151ad8eb75e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44791 } }
I20260926 22:49:16.612133 19290 raft_consensus.cc:374] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:16.612160 19290 raft_consensus.cc:729] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daceeb1ec01a41708d2cd151ad8eb75e, State: Initialized, Role: FOLLOWER
I20260926 22:49:16.612296 19290 consensus_queue.cc:260] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daceeb1ec01a41708d2cd151ad8eb75e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44791 } }
I20260926 22:49:16.612376 19290 raft_consensus.cc:388] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:16.612412 19290 raft_consensus.cc:482] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:16.612394 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19223
I20260926 22:49:16.612461 19290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:16.612493 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:16.612669 13904 external_mini_cluster.cc:1428] Setting key fc61486d29e89b08499a6219800f3dcd
I20260926 22:49:16.613063 19290 raft_consensus.cc:504] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daceeb1ec01a41708d2cd151ad8eb75e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44791 } }
I20260926 22:49:16.613165 19290 leader_election.cc:302] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daceeb1ec01a41708d2cd151ad8eb75e; no voters:
I20260926 22:49:16.613317 19290 leader_election.cc:288] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:16.613335 19295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:16.613435 19295 raft_consensus.cc:686] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [term 1 LEADER]: Becoming Leader. State: Replica: daceeb1ec01a41708d2cd151ad8eb75e, State: Running, Role: LEADER
I20260926 22:49:16.613513 19295 consensus_queue.cc:237] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daceeb1ec01a41708d2cd151ad8eb75e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44791 } }
I20260926 22:49:16.613562 19290 sys_catalog.cc:564] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:16.613878 19297 sys_catalog.cc:455] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daceeb1ec01a41708d2cd151ad8eb75e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daceeb1ec01a41708d2cd151ad8eb75e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44791 } } }
I20260926 22:49:16.613955 19297 sys_catalog.cc:458] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:16.614256 19296 sys_catalog.cc:455] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [sys.catalog]: SysCatalogTable state changed. Reason: New leader daceeb1ec01a41708d2cd151ad8eb75e. Latest consensus state: current_term: 1 leader_uuid: "daceeb1ec01a41708d2cd151ad8eb75e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daceeb1ec01a41708d2cd151ad8eb75e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44791 } } }
I20260926 22:49:16.614506 19296 sys_catalog.cc:458] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:16.614820 19301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:16.615396 19301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:16.616930 19301 catalog_manager.cc:1347] Generated new cluster ID: 8f5ce199306f42c7966d1591665ff482
I20260926 22:49:16.616986 19301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:16.618052 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.613097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.13.148.62:44791)
0926 22:49:16.613315 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:16.613322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:16.613346 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:49:16.613460 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:16.613467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:16.613549 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:16.613921 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:16.614013 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.615354 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.616284 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.616765 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.617165 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.617491 (+ 326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:16.617495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:16.617495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:16.617646 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:16.617676 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:16.617685 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:16.617776 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:16.617828 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 22:49:16.617902 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260926 22:49:16.618806 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:49:16.646956 19301 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:16.647682 19301 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:16.651786 19301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daceeb1ec01a41708d2cd151ad8eb75e: Generated new TSK 0
I20260926 22:49:16.651962 19301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:49:16.667102 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.664070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.13.148.62:44791)
0926 22:49:16.664124 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:16.664126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:16.664235 (+ 109us) server_negotiation.cc:407] Connection header received
0926 22:49:16.664305 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:16.664306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:16.664319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:16.664380 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0926 22:49:16.664508 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.664989 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.665762 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.665959 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.666011 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.666178 (+ 167us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 22:49:16.666915 (+ 737us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35031/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 22:49:16.667035 (+ 120us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35031/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 22:49:16.667161 19294 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35031/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 22:49:16.667187 19319 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:16.664017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:44791 (local address 127.0.0.1:53084)
0926 22:49:16.664174 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:16.664183 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:16.664246 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:16.664383 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:16.664384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:16.664386 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 22:49:16.664452 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:16.664454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.665077 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.665078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:16.665667 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:16.665669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.666024 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.666026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:16.666089 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.666094 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 22:49:16.667001 (+ 907us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35031/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 22:49:16.667089 (+ 88us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.62:44791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35031/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260926 22:49:16.667609 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19223
E20260926 22:49:16.677680 19320 webserver.cc:533] Webserver: sslize failed:
E20260926 22:49:16.677759 19320 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T22:49:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T22:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:16Z Disabled control of system clock
I20260926 22:49:16.711167 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:37999
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:34953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:37999
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/testchainca.cert with env {}
W20260926 22:49:16.788628 19324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:16.788836 19324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:16.788863 19324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:16.790050 19324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:16.790089 19324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:16.790104 19324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:16.790120 19324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:16.790134 19324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 22:49:16.790150 19324 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 22:49:16.790166 19324 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 22:49:16.791644 19324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:37999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:16.791887 19324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:16.792083 19324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:16.796479 19329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:16.796523 19332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:16.796667 19324 server_base.cc:1034] running on GCE node
W20260926 22:49:16.796682 19330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:16.796907 19324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:16.797174 19324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:16.798311 19324 hybrid_clock.cc:648] HybridClock initialized: now 1790462956798295 us; error 28 us; skew 500 ppm
I20260926 22:49:16.799432 19324 webserver.cc:460] Webserver started at http://127.13.148.62:34403/ using document root <none> and password file <none>
I20260926 22:49:16.799642 19324 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:16.799690 19324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:16.799803 19324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:16.800793 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "45071487fda14966b37c0602f6335175"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "64b493aa618e9d9d01944f2b10db6ba3"
server_key_iv: "a85bbc07810cdbd4d6462b5b3faaf802"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.801101 19324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "45071487fda14966b37c0602f6335175"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "64b493aa618e9d9d01944f2b10db6ba3"
server_key_iv: "a85bbc07810cdbd4d6462b5b3faaf802"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.802305 19324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:16.802911 19338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:16.803047 19324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:16.803122 19324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "45071487fda14966b37c0602f6335175"
format_stamp: "Formatted at 2026-09-26 22:49:16 on dist-test-slave-98hb"
server_key: "64b493aa618e9d9d01944f2b10db6ba3"
server_key_iv: "a85bbc07810cdbd4d6462b5b3faaf802"
server_key_version: "encryptionkey@0"
I20260926 22:49:16.803177 19324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:16.818186 19324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:16.818447 19324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:16.818562 19324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:16.822057 19324 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:37999
I20260926 22:49:16.822090 19390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:37999 every 8 connection(s)
I20260926 22:49:16.822396 19324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:16.823268 19391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:16.824671 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19324
I20260926 22:49:16.824793 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:16.824980 13904 external_mini_cluster.cc:1428] Setting key 4e9eb9804ba4b7b72bbe65013af14189
I20260926 22:49:16.825724 19391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175: Bootstrap starting.
I20260926 22:49:16.826301 19391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:16.826620 19391 log.cc:824] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:16.827215 19391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175: No bootstrap required, opened a new log
I20260926 22:49:16.829356 19391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45071487fda14966b37c0602f6335175" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37999 } }
I20260926 22:49:16.829521 19391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:16.829559 19391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45071487fda14966b37c0602f6335175, State: Initialized, Role: FOLLOWER
I20260926 22:49:16.829679 19391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45071487fda14966b37c0602f6335175" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37999 } }
I20260926 22:49:16.829807 19391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:16.829844 19391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:16.829890 19391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:16.830264 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.825465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.13.148.62:37999)
0926 22:49:16.825642 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:16.825649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:16.825672 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:49:16.826164 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:16.826172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:16.826236 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:16.826362 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:16.826575 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.827457 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.828473 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.828852 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.828928 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.829551 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:16.829555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:16.829556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:16.829720 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:16.829799 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:16.829815 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:16.830030 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:16.830055 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 22:49:16.830113 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260926 22:49:16.830461 19391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45071487fda14966b37c0602f6335175" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37999 } }
I20260926 22:49:16.830561 19391 leader_election.cc:302] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45071487fda14966b37c0602f6335175; no voters:
I20260926 22:49:16.830701 19391 leader_election.cc:288] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:16.830745 19396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:16.830871 19396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [term 1 LEADER]: Becoming Leader. State: Replica: 45071487fda14966b37c0602f6335175, State: Running, Role: LEADER
I20260926 22:49:16.830924 19391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:16.830947 19396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45071487fda14966b37c0602f6335175" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37999 } }
I20260926 22:49:16.831176 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:49:16.831228 19396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45071487fda14966b37c0602f6335175. Latest consensus state: current_term: 1 leader_uuid: "45071487fda14966b37c0602f6335175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45071487fda14966b37c0602f6335175" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37999 } } }
I20260926 22:49:16.831290 19396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:16.832067 19398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45071487fda14966b37c0602f6335175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45071487fda14966b37c0602f6335175" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 37999 } } }
I20260926 22:49:16.832232 19398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175 [sys.catalog]: This master's current role is: LEADER
W20260926 22:49:16.832530 19412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:49:16.832581 19412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:49:16.832634 19401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:16.832798 19401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:16.834076 19401 catalog_manager.cc:1347] Generated new cluster ID: 3145017d4e82411e94a4d57af3420aeb
I20260926 22:49:16.834120 19401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:16.847807 19401 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:16.848281 19401 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:16.852532 19401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45071487fda14966b37c0602f6335175: Generated new TSK 0
I20260926 22:49:16.852686 19401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:49:16.936643 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:16.860178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.13.148.62:37999)
0926 22:49:16.860233 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:16.860235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:16.860311 (+ 76us) server_negotiation.cc:407] Connection header received
0926 22:49:16.860388 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:16.860389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:16.860398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:16.860451 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0926 22:49:16.860579 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.861052 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.862174 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.862356 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.862402 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.862551 (+ 149us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 22:49:16.936440 (+ 73889us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 22:49:16.936585 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56936: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 22:49:16.936712 19394 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56936: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 22:49:16.936841 19420 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:16.860124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:37999 (local address 127.0.0.1:56936)
0926 22:49:16.860256 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:16.860267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:16.860327 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:16.860462 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:16.860463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:16.860465 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 22:49:16.860524 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:16.860526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.861110 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.861111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:16.862106 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:16.862109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:16.862418 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:16.862419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:16.862476 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:16.862481 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 22:49:16.936582 (+ 74101us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 22:49:16.936739 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.148.62:37999: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":86,"threads_started":1}
I20260926 22:49:16.937130 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19324
2026-09-26T22:49:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (249 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (84 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:17 dist-test-slave-98hb krb5kdc[19432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:17 dist-test-slave-98hb krb5kdc[19432](info): set up 2 sockets
Sep 26 22:49:17 dist-test-slave-98hb krb5kdc[19432](info): commencing operation
krb5kdc: starting...
W20260926 22:49:18.050006 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:18.159762 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:43889
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:18.236666 19445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:18.236867 19445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:18.236887 19445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:18.238049 19445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:18.238089 19445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:18.238101 19445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:18.238112 19445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:18.238123 19445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:18.239550 19445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:43889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:18.239768 19445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:18.239971 19445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:49:18.244462 19445 server_base.cc:1034] running on GCE node
W20260926 22:49:18.244427 19450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:18.244432 19451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:18.244442 19453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:18.244833 19445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:18.245095 19445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:18.246229 19445 hybrid_clock.cc:648] HybridClock initialized: now 1790462958246216 us; error 29 us; skew 500 ppm
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.248458 19445 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:18.248867 19445 webserver.cc:460] Webserver started at http://127.13.148.62:35799/ using document root <none> and password file <none>
I20260926 22:49:18.249076 19445 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:18.249122 19445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:18.249244 19445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:18.250306 19445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "c8c30dbb17e24ff7a44df18551585fb5"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "28486917f73850025dd2cfc55cfab3e6"
server_key_iv: "ffe367d9208782f1d6908820af2cce99"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.250624 19445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "c8c30dbb17e24ff7a44df18551585fb5"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "28486917f73850025dd2cfc55cfab3e6"
server_key_iv: "ffe367d9208782f1d6908820af2cce99"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.251796 19445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:18.252421 19460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.252565 19445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:18.252636 19445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "c8c30dbb17e24ff7a44df18551585fb5"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "28486917f73850025dd2cfc55cfab3e6"
server_key_iv: "ffe367d9208782f1d6908820af2cce99"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.252693 19445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:18.264092 19445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:18.265820 19445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:18.265944 19445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:18.269441 19445 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:43889
I20260926 22:49:18.269486 19512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:43889 every 8 connection(s)
I20260926 22:49:18.269865 19445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:18.270666 19513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:18.272782 19513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5: Bootstrap starting.
I20260926 22:49:18.273290 19513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:18.273511 19513 log.cc:824] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:18.273932 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19445
I20260926 22:49:18.274039 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:18.274181 19513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5: No bootstrap required, opened a new log
I20260926 22:49:18.274225 13904 external_mini_cluster.cc:1428] Setting key 0262433ddd127a2877f8e5ef76d099cc
I20260926 22:49:18.276312 19513 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c30dbb17e24ff7a44df18551585fb5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43889 } }
I20260926 22:49:18.276448 19513 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:18.276466 19513 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8c30dbb17e24ff7a44df18551585fb5, State: Initialized, Role: FOLLOWER
I20260926 22:49:18.276568 19513 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c30dbb17e24ff7a44df18551585fb5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43889 } }
I20260926 22:49:18.276629 19513 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:18.276649 19513 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:18.276693 19513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:18.277179 19513 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c30dbb17e24ff7a44df18551585fb5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43889 } }
I20260926 22:49:18.277269 19513 leader_election.cc:302] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8c30dbb17e24ff7a44df18551585fb5; no voters:
I20260926 22:49:18.277415 19513 leader_election.cc:288] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:18.277451 19518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:18.277567 19518 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [term 1 LEADER]: Becoming Leader. State: Replica: c8c30dbb17e24ff7a44df18551585fb5, State: Running, Role: LEADER
I20260926 22:49:18.277659 19513 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:18.277642 19518 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c30dbb17e24ff7a44df18551585fb5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43889 } }
I20260926 22:49:18.278020 19518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8c30dbb17e24ff7a44df18551585fb5. Latest consensus state: current_term: 1 leader_uuid: "c8c30dbb17e24ff7a44df18551585fb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c30dbb17e24ff7a44df18551585fb5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43889 } } }
I20260926 22:49:18.278105 19518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:18.278033 19520 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8c30dbb17e24ff7a44df18551585fb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8c30dbb17e24ff7a44df18551585fb5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 43889 } } }
I20260926 22:49:18.278244 19520 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:18.278313 19525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:18.278808 19525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:18.280226 19525 catalog_manager.cc:1347] Generated new cluster ID: 394425a3f8f548bb9620751af365e188
I20260926 22:49:18.280277 19525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:18.284344 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.274691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.13.148.62:43889)
0926 22:49:18.274853 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.274860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.274884 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:49:18.275400 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.275407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.275452 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.275557 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.276293 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.277547 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.278576 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.279242 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.279338 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.281715 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:18.281718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:18.281719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:18.281766 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:18.283157 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.283468 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.283470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.283471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.283540 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.283676 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.283677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.283677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.283876 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:18.283978 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:18.284070 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 22:49:18.284158 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260926 22:49:18.297839 19525 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:18.298334 19525 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:18.308763 19525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8c30dbb17e24ff7a44df18551585fb5: Generated new TSK 0
I20260926 22:49:18.308905 19525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:18.338450 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43889
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:18.415688 19539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:18.415884 19539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:18.415904 19539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:18.417042 19539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:18.417078 19539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:18.417102 19539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:18.418522 19539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:18.418738 19539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:18.418934 19539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:18.423460 19545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:18.423478 19539 server_base.cc:1034] running on GCE node
W20260926 22:49:18.423448 19544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:18.423432 19547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:18.423756 19539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:18.423991 19539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:18.425124 19539 hybrid_clock.cc:648] HybridClock initialized: now 1790462958425113 us; error 26 us; skew 500 ppm
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.427452 19539 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:18.427814 19539 webserver.cc:460] Webserver started at http://127.13.148.1:38951/ using document root <none> and password file <none>
I20260926 22:49:18.427978 19539 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:18.428014 19539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:18.428100 19539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:18.429059 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "7af6ba53c6464adebfa6a8c0c13cec21"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "0c56163c79636b67410a8c44900582c8"
server_key_iv: "66aa3ef42d6533c74f986933807ed176"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.429349 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "7af6ba53c6464adebfa6a8c0c13cec21"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "0c56163c79636b67410a8c44900582c8"
server_key_iv: "66aa3ef42d6533c74f986933807ed176"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.430523 19539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:18.431173 19554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.431365 19539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:18.431449 19539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "7af6ba53c6464adebfa6a8c0c13cec21"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "0c56163c79636b67410a8c44900582c8"
server_key_iv: "66aa3ef42d6533c74f986933807ed176"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.431516 19539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:18.447486 19539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:18.449627 19539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:18.449775 19539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:18.450002 19539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:18.450295 19539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:18.450326 19539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.450358 19539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:18.450380 19539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.455812 19539 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:43519
I20260926 22:49:18.455853 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:43519 every 8 connection(s)
I20260926 22:49:18.456177 19539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:18.462611 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19539
I20260926 22:49:18.462708 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:18.462883 13904 external_mini_cluster.cc:1428] Setting key 267c3c165349414d6b20a66eba2fa8e2
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:18.468511 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.456855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:60697 (local address 127.13.148.62:43889)
0926 22:49:18.456903 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.456905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.457155 (+ 250us) server_negotiation.cc:407] Connection header received
0926 22:49:18.458104 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.458105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.458116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.458163 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.458742 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.459241 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.460155 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.460347 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.460391 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.465931 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:18.465932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:18.465932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:18.465941 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:18.467258 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.467630 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.467630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.467631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.467692 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.467909 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.467910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.467910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.468044 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:18.468112 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:18.468332 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 22:49:18.468415 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:49:18.468593 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.456834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43889 (local address 127.13.148.1:60697)
0926 22:49:18.457017 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:18.457065 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:18.458012 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:18.458229 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:18.458232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:18.458481 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:18.458673 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:18.458677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.459302 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.459302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:18.460093 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:18.460095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.460401 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.460402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:18.460496 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.464190 (+ 3694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:18.464201 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:18.465849 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:18.467337 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:18.467343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:18.467344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:18.467557 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:18.467765 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:18.467766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:18.467766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:18.467857 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:18.468099 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:18.468102 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:18.468212 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 22:49:18.468295 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:49:18.468415 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260926 22:49:18.469043 19668 heartbeater.cc:347] Connected to a master server at 127.13.148.62:43889
I20260926 22:49:18.469137 19668 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:18.469312 19668 heartbeater.cc:511] Master 127.13.148.62:43889 requested a full tablet report, sending...
I20260926 22:49:18.469847 19477 ts_manager.cc:194] Registered new tserver with Master: 7af6ba53c6464adebfa6a8c0c13cec21 (127.13.148.1:43519)
I20260926 22:49:18.470746 19477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:60697
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:18.487881 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43889
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:18.565574 19673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:18.565877 19673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:18.565912 19673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:18.567215 19673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:18.567291 19673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:18.567323 19673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:18.569128 19673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:18.569446 19673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:18.569806 19673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:18.575416 19679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:18.575651 19678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:18.575678 19681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:18.575652 19673 server_base.cc:1034] running on GCE node
I20260926 22:49:18.576009 19673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:18.576295 19673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:18.577448 19673 hybrid_clock.cc:648] HybridClock initialized: now 1790462958577431 us; error 38 us; skew 500 ppm
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.580250 19673 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:18.580756 19673 webserver.cc:460] Webserver started at http://127.13.148.2:41707/ using document root <none> and password file <none>
I20260926 22:49:18.580982 19673 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:18.581027 19673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:18.581138 19673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:18.582252 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "0242cd4e071a4430a47137b8e8403b0d"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "e9b081a4bf95b9ac38e6f073552f7160"
server_key_iv: "75bace07fd055366199709f1e283bf3e"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.582566 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "0242cd4e071a4430a47137b8e8403b0d"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "e9b081a4bf95b9ac38e6f073552f7160"
server_key_iv: "75bace07fd055366199709f1e283bf3e"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.583809 19673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:18.584497 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.584666 19673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:18.584746 19673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "0242cd4e071a4430a47137b8e8403b0d"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "e9b081a4bf95b9ac38e6f073552f7160"
server_key_iv: "75bace07fd055366199709f1e283bf3e"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.584810 19673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:18.595018 19673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:18.596829 19673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:18.596935 19673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:18.597111 19673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:18.597350 19673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:18.597378 19673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.597397 19673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:18.597409 19673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.602741 19673 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:40729
I20260926 22:49:18.602813 19801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:40729 every 8 connection(s)
I20260926 22:49:18.603158 19673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:18.611485 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.603801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:48091 (local address 127.13.148.62:43889)
0926 22:49:18.603842 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.603843 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.604050 (+ 207us) server_negotiation.cc:407] Connection header received
0926 22:49:18.604965 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.604967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.604979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.605029 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.605640 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.606192 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.607145 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.607343 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.607396 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.609253 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:18.609255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:18.609255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:18.609262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:18.610314 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.610656 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.610657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.610657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.610707 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.610914 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.610914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.610915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.611016 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:18.611070 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:18.611322 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 22:49:18.611396 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:49:18.611644 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.603788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43889 (local address 127.13.148.2:48091)
0926 22:49:18.603938 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:18.603972 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:18.604862 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:18.605101 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:18.605105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:18.605368 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:18.605552 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:18.605556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.606279 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.606280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:18.607066 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:18.607068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.607409 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.607410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:18.607527 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.607910 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:18.607918 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:18.609169 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:18.610382 (+ 1213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:18.610388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:18.610390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:18.610590 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:18.610759 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:18.610760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:18.610760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:18.610859 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:18.611069 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:18.611075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:18.611200 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 22:49:18.611293 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 22:49:18.611442 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260926 22:49:18.612139 19802 heartbeater.cc:347] Connected to a master server at 127.13.148.62:43889
I20260926 22:49:18.612156 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19673
I20260926 22:49:18.612236 19802 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:18.612248 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:18.612401 19802 heartbeater.cc:511] Master 127.13.148.62:43889 requested a full tablet report, sending...
I20260926 22:49:18.612417 13904 external_mini_cluster.cc:1428] Setting key c39aab8e95bf938612ccda597f055b4a
I20260926 22:49:18.612739 19477 ts_manager.cc:194] Registered new tserver with Master: 0242cd4e071a4430a47137b8e8403b0d (127.13.148.2:40729)
I20260926 22:49:18.613163 19477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:48091
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:18.634732 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43889
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:18.713794 19807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:18.714008 19807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:18.714039 19807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:18.715193 19807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:18.715235 19807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:18.715276 19807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:18.716737 19807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:43889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:18.716972 19807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:18.717175 19807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:18.721975 19815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:18.722028 19812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:18.722064 19807 server_base.cc:1034] running on GCE node
W20260926 22:49:18.722028 19813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:18.722399 19807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:18.722625 19807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:18.723773 19807 hybrid_clock.cc:648] HybridClock initialized: now 1790462958723762 us; error 28 us; skew 500 ppm
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.726161 19807 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:18.726532 19807 webserver.cc:460] Webserver started at http://127.13.148.3:46395/ using document root <none> and password file <none>
I20260926 22:49:18.726702 19807 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:18.726740 19807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:18.726837 19807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:18.727821 19807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "b81e1de81623450e91ec0fcae4ecfc77"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "49dae7ac546e8c622e6cfac737235941"
server_key_iv: "1ccf3580675a695213a1161747ccfd48"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.728108 19807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "b81e1de81623450e91ec0fcae4ecfc77"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "49dae7ac546e8c622e6cfac737235941"
server_key_iv: "1ccf3580675a695213a1161747ccfd48"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.729370 19807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:18.730105 19822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.730254 19807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.730340 19807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "b81e1de81623450e91ec0fcae4ecfc77"
format_stamp: "Formatted at 2026-09-26 22:49:18 on dist-test-slave-98hb"
server_key: "49dae7ac546e8c622e6cfac737235941"
server_key_iv: "1ccf3580675a695213a1161747ccfd48"
server_key_version: "encryptionkey@0"
I20260926 22:49:18.730401 19807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:18.738191 19807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:18.740226 19807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:18.740355 19807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:18.740556 19807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:18.740839 19807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:18.740870 19807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.740897 19807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:18.740914 19807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:18.746289 19807 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:42195
I20260926 22:49:18.746325 19935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:42195 every 8 connection(s)
I20260926 22:49:18.746622 19807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:18.748787 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19807
I20260926 22:49:18.748872 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:18.749042 13904 external_mini_cluster.cc:1428] Setting key 63f0cd867e44a6480446d0ed1d09736b
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:18.755617 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.747289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:49399 (local address 127.13.148.62:43889)
0926 22:49:18.747342 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.747344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.747542 (+ 198us) server_negotiation.cc:407] Connection header received
0926 22:49:18.748475 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.748477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.748488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.748540 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.749195 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.749814 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.750795 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.750999 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.751054 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.753100 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:18.753102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:18.753102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:18.753110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:18.754335 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.754729 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.754730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.754731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.754806 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.755022 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.755023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.755023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.755137 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:18.755193 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:18.755448 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 22:49:18.755525 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:49:18.755800 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.747269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:43889 (local address 127.13.148.3:49399)
0926 22:49:18.747427 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:18.747462 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:18.748362 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:18.748616 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:18.748620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:18.748889 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:18.749105 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:18.749111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.749896 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.749897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:18.750709 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:18.750712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.751067 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.751068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:18.751192 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.751648 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:18.751657 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:18.753008 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:18.754409 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:18.754416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:18.754418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:18.754669 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:18.754858 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:18.754859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:18.754859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:18.754968 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:18.755190 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:18.755196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:18.755332 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 22:49:18.755422 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 22:49:18.755577 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260926 22:49:18.756253 19936 heartbeater.cc:347] Connected to a master server at 127.13.148.62:43889
I20260926 22:49:18.756349 19936 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:18.756512 19936 heartbeater.cc:511] Master 127.13.148.62:43889 requested a full tablet report, sending...
I20260926 22:49:18.756870 19477 ts_manager.cc:194] Registered new tserver with Master: b81e1de81623450e91ec0fcae4ecfc77 (127.13.148.3:42195)
I20260926 22:49:18.757232 19477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:49399
I20260926 22:49:18.760818 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/test-admin.keytab.
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.775744 13904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 22:49:18.777295 19943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:18.776547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46680 (local address 127.13.148.1:43519)
0926 22:49:18.776696 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.776702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.776710 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:18.776739 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.776741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.776753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.776815 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.777201 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46680: BlockingRecv error: recv got EOF from 127.0.0.1:46680 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
W20260926 22:49:18.777316 13904 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5cc)
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.778326 13904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.779172 13904 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:49:18.786350 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.779659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.13.148.1:43519)
0926 22:49:18.779702 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.779704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.779746 (+ 42us) server_negotiation.cc:407] Connection header received
0926 22:49:18.779815 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.779816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.779829 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.779869 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.780219 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.780923 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.781625 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.781996 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.782056 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.783969 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:18.783977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:18.783978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:18.783985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:18.785307 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.785598 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.785601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.785601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.785653 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.785829 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.785830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.785830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.785963 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:18.786092 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:18.786166 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 22:49:18.786251 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 22:49:18.787954 19943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:18.787468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46696 (local address 127.13.148.1:43519)
0926 22:49:18.787515 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.787517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.787540 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:49:18.787606 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.787607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.787617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.787656 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.787913 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46696: BlockingRecv error: recv got EOF from 127.0.0.1:46696 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 22:49:18.787997 13904 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5cc' of type FILE
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.788862 13904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.789601 13904 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:49:18.795375 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.790057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.13.148.1:43519)
0926 22:49:18.790112 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.790114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.790149 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:49:18.790185 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.790186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.790195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.790231 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.790525 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.790982 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.791581 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.791760 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.791800 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.793455 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:18.793456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:18.793456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:18.793462 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:18.794547 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.794847 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.794848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.794848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.794892 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.795020 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.795020 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.795020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.795096 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:18.795153 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:18.795226 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 22:49:18.795295 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 22:49:18.796722 19943 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:18.796233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.13.148.1:43519)
0926 22:49:18.796288 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.796289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.796310 (+ 21us) server_negotiation.cc:407] Connection header received
0926 22:49:18.796372 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.796373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.796380 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.796430 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.796678 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46700: BlockingRecv error: recv got EOF from 127.0.0.1:46700 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 22:49:18.796784 13904 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790462935413609-13904-0/krb5kdc/krb5cc' of type FILE
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.797605 13904 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:18.798354 13904 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:49:18.803906 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:18.798776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.13.148.1:43519)
0926 22:49:18.798826 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:18.798827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:18.798837 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:49:18.798890 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:18.798890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:18.798897 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:18.798932 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:18.799209 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.799661 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.800284 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:18.800449 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:18.800492 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:18.802178 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:18.802179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:18.802179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:18.802185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:18.803152 (+ 967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.803432 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.803432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.803433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.803475 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:18.803589 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:18.803589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:18.803590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:18.803661 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:18.803714 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:18.803786 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 22:49:18.803835 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:49:18.804464 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19539
I20260926 22:49:18.808820 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19673
I20260926 22:49:18.812623 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19807
I20260926 22:49:18.816170 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19445
2026-09-26T22:49:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1710 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19956](info): set up 2 sockets
Sep 26 22:49:18 dist-test-slave-98hb krb5kdc[19956](info): commencing operation
krb5kdc: starting...
W20260926 22:49:19.766008 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:19 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.62.keytab.
Entry for principal oryx/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.62.keytab.
I20260926 22:49:19.873579 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40093
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.62.keytab
--principal=oryx/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:19.950162 19969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:19.950356 19969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:19.950377 19969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:19.951454 19969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:19.951486 19969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:19.951498 19969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:19.951509 19969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:19.951519 19969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:19.952891 19969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.62.keytab
--principal=oryx/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:19.953071 19969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:19.953238 19969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:19.957625 19977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:19.957607 19975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:19.957640 19969 server_base.cc:1034] running on GCE node
W20260926 22:49:19.957615 19974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:19.958016 19969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:19.958273 19969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:19.959400 19969 hybrid_clock.cc:648] HybridClock initialized: now 1790462959959390 us; error 27 us; skew 500 ppm
Sep 26 22:49:19 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462959, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:19.961544 19969 init.cc:377] Logged in from keytab as oryx/127.13.148.62@KRBTEST.COM (short username oryx)
I20260926 22:49:19.961977 19969 webserver.cc:460] Webserver started at http://127.13.148.62:37161/ using document root <none> and password file <none>
I20260926 22:49:19.962175 19969 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:19.962217 19969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:19.962333 19969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:19.963294 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "6906a7f3477746d9bee593df5a14375d"
format_stamp: "Formatted at 2026-09-26 22:49:19 on dist-test-slave-98hb"
server_key: "b6b537dc598cd5504c372b8fb1a6dde6"
server_key_iv: "f3c33daa2a184aec87e240da1607dc7c"
server_key_version: "encryptionkey@0"
I20260926 22:49:19.963598 19969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "6906a7f3477746d9bee593df5a14375d"
format_stamp: "Formatted at 2026-09-26 22:49:19 on dist-test-slave-98hb"
server_key: "b6b537dc598cd5504c372b8fb1a6dde6"
server_key_iv: "f3c33daa2a184aec87e240da1607dc7c"
server_key_version: "encryptionkey@0"
I20260926 22:49:19.964759 19969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:49:19.965348 19984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:19.965485 19969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:19.965565 19969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "6906a7f3477746d9bee593df5a14375d"
format_stamp: "Formatted at 2026-09-26 22:49:19 on dist-test-slave-98hb"
server_key: "b6b537dc598cd5504c372b8fb1a6dde6"
server_key_iv: "f3c33daa2a184aec87e240da1607dc7c"
server_key_version: "encryptionkey@0"
I20260926 22:49:19.965616 19969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:19.995690 19969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:19.997810 19969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:19.997917 19969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:20.001327 19969 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40093
I20260926 22:49:20.001370 20036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40093 every 8 connection(s)
I20260926 22:49:20.001669 19969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:20.002624 20037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:20.004813 20037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: Bootstrap starting.
I20260926 22:49:20.005343 20037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:20.005585 20037 log.cc:824] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:20.006261 20037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: No bootstrap required, opened a new log
I20260926 22:49:20.007740 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 19969
I20260926 22:49:20.007833 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:20.007769 20037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } }
I20260926 22:49:20.007932 20037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:20.007977 20037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6906a7f3477746d9bee593df5a14375d, State: Initialized, Role: FOLLOWER
I20260926 22:49:20.008004 13904 external_mini_cluster.cc:1428] Setting key 9c9f1df673a6ff7a661d01a59b8cf7cc
I20260926 22:49:20.008121 20037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } }
I20260926 22:49:20.008198 20037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:20.008230 20037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:20.008284 20037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:20.009100 20037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } }
I20260926 22:49:20.009195 20037 leader_election.cc:302] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6906a7f3477746d9bee593df5a14375d; no voters:
I20260926 22:49:20.009352 20037 leader_election.cc:288] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:20.009377 20042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:20.009482 20042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 1 LEADER]: Becoming Leader. State: Replica: 6906a7f3477746d9bee593df5a14375d, State: Running, Role: LEADER
I20260926 22:49:20.009560 20042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } }
I20260926 22:49:20.009599 20037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:20.009888 20042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6906a7f3477746d9bee593df5a14375d. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } }
I20260926 22:49:20.009956 20042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:20.010089 20044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } }
I20260926 22:49:20.010162 20044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:20.010280 20046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:20.011029 20046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:20.012506 20046 catalog_manager.cc:1347] Generated new cluster ID: a71ea67d9f974a92ae76bc21db738a4c
I20260926 22:49:20.012552 20046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:20.017472 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.008454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.13.148.62:40093)
0926 22:49:20.008626 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.008635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.008663 (+ 28us) server_negotiation.cc:407] Connection header received
0926 22:49:20.008772 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.008777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.008818 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.008930 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.009604 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.010994 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.011982 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.012455 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.012556 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.014643 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.014647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.014647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.014659 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.016167 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.016469 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.016472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.016472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.016561 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.016727 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.016728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.016728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.016945 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.017092 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.017157 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.017298 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260926 22:49:20.031343 20046 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:20.032052 20046 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:20.041215 20046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: Generated new TSK 0
I20260926 22:49:20.041395 20046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
I20260926 22:49:20.063864 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40093
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:20.141089 20063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:20.141299 20063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:20.141330 20063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:20.142488 20063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:20.142527 20063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:20.142562 20063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:20.143956 20063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:20.144182 20063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:20.144387 20063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:20.149021 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:20.149024 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.149163 20063 server_base.cc:1034] running on GCE node
W20260926 22:49:20.149044 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.149405 20063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:20.149645 20063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:20.150790 20063 hybrid_clock.cc:648] HybridClock initialized: now 1790462960150776 us; error 28 us; skew 500 ppm
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:20.153000 20063 init.cc:377] Logged in from keytab as oryx/127.13.148.1@KRBTEST.COM (short username oryx)
I20260926 22:49:20.153384 20063 webserver.cc:460] Webserver started at http://127.13.148.1:38679/ using document root <none> and password file <none>
I20260926 22:49:20.153582 20063 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:20.153630 20063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:20.153782 20063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:20.154776 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "fb65cccc125342c893b4fd43be439bee"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "bcdceb4e3edd3953645d9f878efe0154"
server_key_iv: "2310800411cb7ee23293696c485d1ee4"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.155089 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "fb65cccc125342c893b4fd43be439bee"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "bcdceb4e3edd3953645d9f878efe0154"
server_key_iv: "2310800411cb7ee23293696c485d1ee4"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.156265 20063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:49:20.156896 20078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.157025 20063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:20.157101 20063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "fb65cccc125342c893b4fd43be439bee"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "bcdceb4e3edd3953645d9f878efe0154"
server_key_iv: "2310800411cb7ee23293696c485d1ee4"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.157160 20063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:20.168955 20063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:20.170816 20063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:20.170951 20063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:20.171159 20063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:20.171496 20063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:20.171536 20063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.171567 20063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:20.171590 20063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.176887 20063 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:43653
I20260926 22:49:20.176921 20191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:43653 every 8 connection(s)
I20260926 22:49:20.177213 20063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:20.178267 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 20063
I20260926 22:49:20.178349 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:20.178493 13904 external_mini_cluster.cc:1428] Setting key 96f6c16414f713794e77b5ada4d42b7e
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260926 22:49:20.187203 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.177830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38529 (local address 127.13.148.62:40093)
0926 22:49:20.177873 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.177875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.178154 (+ 279us) server_negotiation.cc:407] Connection header received
0926 22:49:20.178685 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.178686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.178703 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.178751 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.179705 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.180400 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.181770 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.181982 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.182030 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.184510 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.184512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.184512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.184522 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.185888 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.186265 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.186266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.186266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.186321 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.186561 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.186562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.186562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.186683 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.186745 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.187013 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.187109 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:20.187386 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.177835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.1:38529)
0926 22:49:20.178036 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.178071 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.178594 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.178853 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.178858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.179431 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.179616 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.179621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.180500 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.180501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.181673 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.181677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.182057 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.182058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.182198 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.182765 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.182777 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.184397 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.185956 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.185965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.185967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.186189 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.186377 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.186377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.186378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.186503 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.186735 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.186742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.186892 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.186987 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.187153 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260926 22:49:20.187839 20192 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40093
I20260926 22:49:20.187927 20192 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:20.188098 20192 heartbeater.cc:511] Master 127.13.148.62:40093 requested a full tablet report, sending...
I20260926 22:49:20.188659 20001 ts_manager.cc:194] Registered new tserver with Master: fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653)
I20260926 22:49:20.189296 20001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:38529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
I20260926 22:49:20.209823 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40093
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:20.289366 20197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:20.289561 20197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:20.289588 20197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:20.290729 20197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:20.290772 20197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:20.290797 20197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:20.292200 20197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:20.292421 20197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:20.292639 20197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:20.297232 20205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:20.297302 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:20.297412 20202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.297338 20197 server_base.cc:1034] running on GCE node
I20260926 22:49:20.297662 20197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:20.297910 20197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:20.299054 20197 hybrid_clock.cc:648] HybridClock initialized: now 1790462960299042 us; error 26 us; skew 500 ppm
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:20.301378 20197 init.cc:377] Logged in from keytab as oryx/127.13.148.2@KRBTEST.COM (short username oryx)
I20260926 22:49:20.301832 20197 webserver.cc:460] Webserver started at http://127.13.148.2:36015/ using document root <none> and password file <none>
I20260926 22:49:20.302042 20197 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:20.302091 20197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:20.302210 20197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:20.303213 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "d2d0177c27c144b9b103823eb2cab339"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "0ce50fdfb55c5b552a471c6865de71d4"
server_key_iv: "82dca725bd31c6859fdd0733d2a11870"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.303526 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "d2d0177c27c144b9b103823eb2cab339"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "0ce50fdfb55c5b552a471c6865de71d4"
server_key_iv: "82dca725bd31c6859fdd0733d2a11870"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.304754 20197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:20.305374 20212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.305536 20197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:20.305624 20197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "d2d0177c27c144b9b103823eb2cab339"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "0ce50fdfb55c5b552a471c6865de71d4"
server_key_iv: "82dca725bd31c6859fdd0733d2a11870"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.305681 20197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:20.316956 20197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:20.318571 20197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:20.318696 20197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:20.318883 20197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:20.319161 20197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:20.319195 20197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.319221 20197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:20.319237 20197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.324352 20197 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:33849
I20260926 22:49:20.324420 20325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:33849 every 8 connection(s)
I20260926 22:49:20.324695 20197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:20.332695 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.325324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:60091 (local address 127.13.148.62:40093)
0926 22:49:20.325373 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.325375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.325583 (+ 208us) server_negotiation.cc:407] Connection header received
0926 22:49:20.326053 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.326054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.326098 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.326149 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.326980 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.327477 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.328400 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.328590 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.328643 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.330616 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.330617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.330617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.330624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.331723 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.332048 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.332049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.332049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.332091 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.332256 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.332257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.332257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.332351 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.332395 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.332582 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.332634 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:20.332856 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.325304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.2:60091)
0926 22:49:20.325473 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.325507 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.325977 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.326213 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.326216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.326734 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.326914 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.326918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.327546 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.327546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.328321 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.328323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.328659 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.328660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.328754 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.329192 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.329200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.330493 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.331787 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.331792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.331793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.331994 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.332134 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.332134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.332135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.332217 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.332394 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.332396 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.332497 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.332569 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.332676 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 22:49:20.333284 20326 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40093
I20260926 22:49:20.333359 20326 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:20.333489 20326 heartbeater.cc:511] Master 127.13.148.62:40093 requested a full tablet report, sending...
I20260926 22:49:20.333810 20001 ts_manager.cc:194] Registered new tserver with Master: d2d0177c27c144b9b103823eb2cab339 (127.13.148.2:33849)
I20260926 22:49:20.334172 20001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:60091
I20260926 22:49:20.334795 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 20197
I20260926 22:49:20.334877 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:20.335017 13904 external_mini_cluster.cc:1428] Setting key 26cf25f59f76717f006d36424ff45bfe
WARNING: no policy specified for oryx/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
I20260926 22:49:20.356513 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40093
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:20.435448 20331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:20.435637 20331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:20.435660 20331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:20.436739 20331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:20.436774 20331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:20.436797 20331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:20.438167 20331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:20.438347 20331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:20.438515 20331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:20.443011 20339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:20.443014 20337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:20.443040 20336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.443112 20331 server_base.cc:1034] running on GCE node
I20260926 22:49:20.443403 20331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:20.443589 20331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:20.444728 20331 hybrid_clock.cc:648] HybridClock initialized: now 1790462960444720 us; error 36 us; skew 500 ppm
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:20.447077 20331 init.cc:377] Logged in from keytab as oryx/127.13.148.3@KRBTEST.COM (short username oryx)
I20260926 22:49:20.447443 20331 webserver.cc:460] Webserver started at http://127.13.148.3:33117/ using document root <none> and password file <none>
I20260926 22:49:20.447608 20331 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:20.447645 20331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:20.447731 20331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:20.448732 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "53cc0cfba88643bbbfc7d77fbd24fa30"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "f50581177675f81bb988130a23268372"
server_key_iv: "fc72a5fc42b1613bf1d7cc0a78db947a"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.449014 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "53cc0cfba88643bbbfc7d77fbd24fa30"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "f50581177675f81bb988130a23268372"
server_key_iv: "fc72a5fc42b1613bf1d7cc0a78db947a"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.450222 20331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:20.450860 20346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.450990 20331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:20.451068 20331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "53cc0cfba88643bbbfc7d77fbd24fa30"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "f50581177675f81bb988130a23268372"
server_key_iv: "fc72a5fc42b1613bf1d7cc0a78db947a"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.451133 20331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:20.465338 20331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:20.467363 20331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:20.467494 20331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:20.467696 20331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:20.467967 20331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:20.467998 20331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.468024 20331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:20.468039 20331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.473275 20331 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36819
I20260926 22:49:20.473335 20459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36819 every 8 connection(s)
I20260926 22:49:20.473592 20331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:20.480826 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 20331
I20260926 22:49:20.480926 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:20.481120 13904 external_mini_cluster.cc:1428] Setting key df2fab3d5c5fd23193a23920090ca958
I20260926 22:49:20.482043 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.474217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:53561 (local address 127.13.148.62:40093)
0926 22:49:20.474265 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.474267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.474451 (+ 184us) server_negotiation.cc:407] Connection header received
0926 22:49:20.474948 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.474949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.474961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.475007 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.475911 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.476413 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.477366 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.477553 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.477607 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.479564 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.479566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.479566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.479573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.480584 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.481065 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.481066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.481066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.481119 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.481363 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.481364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.481365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.481492 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.481553 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.481874 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.481953 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:49:20.482215 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.474204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.3:53561)
0926 22:49:20.474340 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.474375 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.474857 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.475088 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.475092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.475625 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.475820 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.475824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.476493 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.476494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.477283 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.477285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.477618 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.477619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.477768 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.478163 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.478170 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.479473 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.480679 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.480686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.480688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.480992 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.481174 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.481175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.481176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.481304 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.481550 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.481555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.481706 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.481842 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.482010 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260926 22:49:20.482587 20460 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40093
I20260926 22:49:20.482673 20460 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:20.482844 20460 heartbeater.cc:511] Master 127.13.148.62:40093 requested a full tablet report, sending...
I20260926 22:49:20.483161 20001 ts_manager.cc:194] Registered new tserver with Master: 53cc0cfba88643bbbfc7d77fbd24fa30 (127.13.148.3:36819)
I20260926 22:49:20.483481 20001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:53561
I20260926 22:49:20.492630 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.61.keytab.
Entry for principal oryx/127.13.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.61.keytab.
I20260926 22:49:20.514468 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:34813
--webserver_interface=127.13.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.61.keytab
--principal=oryx/127.13.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40093,127.13.148.61:34813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:20.591648 20465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:20.591852 20465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:20.591872 20465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:20.592980 20465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:20.593020 20465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:20.593034 20465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:20.593044 20465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:20.593058 20465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:20.594511 20465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40093,127.13.148.61:34813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.61.keytab
--principal=oryx/127.13.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:34813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
W20260926 22:49:20.594616 20465 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.148.62:40093,127.13.148.61:34813'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 22:49:20.594765 20465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:20.594965 20465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:20.599324 20473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.599478 20465 server_base.cc:1034] running on GCE node
W20260926 22:49:20.599339 20470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:20.599356 20471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.599730 20465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:20.599992 20465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:20.601122 20465 hybrid_clock.cc:648] HybridClock initialized: now 1790462960601109 us; error 27 us; skew 500 ppm
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:20.603371 20465 init.cc:377] Logged in from keytab as oryx/127.13.148.61@KRBTEST.COM (short username oryx)
I20260926 22:49:20.603730 20465 webserver.cc:460] Webserver started at http://127.13.148.61:35211/ using document root <none> and password file <none>
I20260926 22:49:20.603899 20465 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:20.603935 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:20.604019 20465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:20.604986 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data/instance:
uuid: "6988137b78334268b85f4ee28a34a66d"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "d3d16117a61c6c07a91dd261df92ee8b"
server_key_iv: "804817af6820854fdec3e53abaa7aa65"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.605259 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal/instance:
uuid: "6988137b78334268b85f4ee28a34a66d"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "d3d16117a61c6c07a91dd261df92ee8b"
server_key_iv: "804817af6820854fdec3e53abaa7aa65"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.606415 20465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:20.607004 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.607161 20465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:20.607235 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
uuid: "6988137b78334268b85f4ee28a34a66d"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "d3d16117a61c6c07a91dd261df92ee8b"
server_key_iv: "804817af6820854fdec3e53abaa7aa65"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.607291 20465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:20.620813 20465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:20.622390 20465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:20.622491 20465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:20.625916 20465 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:34813
I20260926 22:49:20.625943 20532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:34813 every 8 connection(s)
I20260926 22:49:20.626248 20465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
I20260926 22:49:20.627099 20533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:20.628479 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 20465
I20260926 22:49:20.628602 20533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.61@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260926 22:49:20.637949 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.629316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.13.148.62:40093)
0926 22:49:20.629356 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.629357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.629519 (+ 162us) server_negotiation.cc:407] Connection header received
0926 22:49:20.629787 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.629789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.629800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.629844 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.630530 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.631299 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.632491 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.632791 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.632865 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.635169 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.635171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.635171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.635178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.636596 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.637013 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.637013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.637014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.637059 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.637278 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.637279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.637279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.637466 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.637539 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.637828 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.637888 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:49:20.638425 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.629257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:34766)
0926 22:49:20.629415 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.629447 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.629701 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.629875 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.629878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.630210 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.630453 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.630459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.631379 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.631380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.632377 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.632380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.632875 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.632876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.632965 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.633393 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.633402 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.635087 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.636673 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.636678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.636679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.636949 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.637110 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.637111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.637111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.637226 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.637533 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.637537 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.637680 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.637801 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.637945 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260926 22:49:20.638425 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.628978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.13.148.61:34813)
0926 22:49:20.629148 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.629157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.629184 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:49:20.629693 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.629700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.629721 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.629832 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.630478 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.631561 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.632508 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.632971 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.633022 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.635607 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.635614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.635615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.635621 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.637181 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.637541 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.637543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.637544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.637596 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.637786 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.637787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.637787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.637926 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.638050 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.638151 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.638211 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260926 22:49:20.639137 20533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.61@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260926 22:49:20.644143 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.639344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.0.0.1:33506)
0926 22:49:20.639385 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.639393 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.639424 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.639509 (+ 85us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.639509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.639605 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.639644 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.639645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.640189 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.640190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.640654 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.640655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.640922 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.640923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.640969 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.641247 (+ 278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.641250 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.642284 (+ 1034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.643508 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.643508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.643509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.643635 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.643760 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.643761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.643761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.643799 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.643941 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.643942 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.643970 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.644012 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.644072 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 22:49:20.644203 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.639370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.13.148.61:34813)
0926 22:49:20.639409 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.639410 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.639418 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:20.639467 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.639468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.639474 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.639505 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.639695 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.640131 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.640707 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.640879 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.640918 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.642349 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.642350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.642350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.642355 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.643453 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.643681 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.643681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.643681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.643721 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.643830 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.643830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.643830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.643905 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.643944 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.644026 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.644110 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":22}
I20260926 22:49:20.645258 20533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Bootstrap starting.
I20260926 22:49:20.645993 20533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:20.646205 20533 log.cc:824] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:20.646793 20533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: No bootstrap required, opened a new log
I20260926 22:49:20.648093 20533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } }
I20260926 22:49:20.648211 20533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:20.648247 20533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6988137b78334268b85f4ee28a34a66d, State: Initialized, Role: FOLLOWER
I20260926 22:49:20.648380 20533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } }
I20260926 22:49:20.648614 20539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } } }
I20260926 22:49:20.648661 20533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:20.648698 20539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:49:20.650126 20550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:49:20.650168 20550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:49:20.651535 20465 master_runner.cc:428] Detected that this master 6988137b78334268b85f4ee28a34a66d is joining an existing cluster
I20260926 22:49:20.651572 20465 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.148.61:34813
I20260926 22:49:20.654242 20539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:20.654315 20539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } }
I20260926 22:49:20.654485 20539 leader_election.cc:288] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6906a7f3477746d9bee593df5a14375d (127.13.148.62:40093)
I20260926 22:49:20.654773 20011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6988137b78334268b85f4ee28a34a66d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6906a7f3477746d9bee593df5a14375d" is_pre_election: true
I20260926 22:49:20.655109 20484 leader_election.cc:302] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 6988137b78334268b85f4ee28a34a66d; no voters: 6906a7f3477746d9bee593df5a14375d
I20260926 22:49:20.655267 20539 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 22:49:20.658510 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.652387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.0.0.1:33514)
0926 22:49:20.652545 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.652555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.652625 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.652757 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.652757 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.652949 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.653046 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.653048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.653939 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.653940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.654504 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.654507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.655057 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.655058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.655119 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.655496 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.655500 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.655829 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.657494 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.657495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.657495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.657678 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.657900 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.657901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.657901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.657947 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.658177 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.658177 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.658209 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.658287 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.658403 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260926 22:49:20.658833 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.652408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.13.148.61:34813)
0926 22:49:20.652441 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.652442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.652620 (+ 178us) server_negotiation.cc:407] Connection header received
0926 22:49:20.652683 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.652684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.652692 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.652746 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.653138 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.653870 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.654574 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.654814 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.654855 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.655904 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.655905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.655905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.655912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.657413 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.657791 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.657792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.657793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.657843 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.658004 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.658004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.658005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.658104 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.658140 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.658322 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.658459 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":17}
I20260926 22:49:20.659361 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.652399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.13.148.62:40093)
0926 22:49:20.652438 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.652440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.652609 (+ 169us) server_negotiation.cc:407] Connection header received
0926 22:49:20.652697 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.652698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.652706 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.652732 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.653413 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.653942 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.655678 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.655968 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.656022 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.656904 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.656905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.656906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.656912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.658137 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.658530 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.658531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.658532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.658587 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.658786 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.658787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.658788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.658918 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.658986 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.659178 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.659287 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:49:20.659361 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.652410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:34780)
0926 22:49:20.652550 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.652558 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.652632 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.652788 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.652788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.653301 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.653341 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.653343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.654016 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.654017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.655599 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.655601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.656025 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.656026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.656111 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.656471 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.656475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.656811 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.658208 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.658209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.658209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.658448 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.658658 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.658658 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.658659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.658718 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.659013 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.659014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.659051 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.659136 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.659234 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462959, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260926 22:49:20.668126 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.661191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.13.148.61:34813)
0926 22:49:20.661226 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.661227 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.661333 (+ 106us) server_negotiation.cc:407] Connection header received
0926 22:49:20.661412 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.661412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.661419 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.661460 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.661917 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.662328 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.663342 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.663505 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.663542 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.665421 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.665422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.665423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.665428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.666695 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.667165 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.667166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.667166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.667273 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.667529 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.667529 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.667530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.667678 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.667742 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.667941 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.668043 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:20.668423 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.661132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.0.0.1:33530)
0926 22:49:20.661237 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.661274 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.661345 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.661466 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.661469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.661801 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.661853 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.661855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.662389 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.662390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.663275 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.663277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.663557 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.663558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.663638 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.663975 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.663979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.665348 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.666781 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.666788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.666790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.667045 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.667357 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.667358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.667358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.667448 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.667736 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.667742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.667808 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.668190 (+ 382us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.668319 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":51,"threads_started":1}
I20260926 22:49:20.669708 20001 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.148.61:34813: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6988137b78334268b85f4ee28a34a66d"
member_type: NON_VOTER
last_known_addr {
host: "127.13.148.61"
port: 34813
}
attrs {
promote: true
}
}
dest_uuid: "6906a7f3477746d9bee593df5a14375d"
cas_config_opid_index: -1
I20260926 22:49:20.670032 20001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } }
I20260926 22:49:20.670490 20560 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6988137b78334268b85f4ee28a34a66d (127.13.148.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } }
I20260926 22:49:20.670724 20507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:20.671418 20507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Refusing update from remote peer 6906a7f3477746d9bee593df5a14375d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 22:49:20.671430 20560 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.148.61:34813
I20260926 22:49:20.671501 20560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } }
I20260926 22:49:20.671554 20560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:20.671633 20465 master.cc:537] Master@127.13.148.61:34813 shutting down...
I20260926 22:49:20.671659 20560 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 22:49:20.672251 19986 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 22:49:20.672654 19986 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d -> Peer 6988137b78334268b85f4ee28a34a66d (127.13.148.61:34813): Couldn't send request to peer 6988137b78334268b85f4ee28a34a66d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 22:49:20.674082 20465 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:20.674155 20465 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:20.674202 20465 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: stopping tablet replica
I20260926 22:49:20.686559 20465 master.cc:552] Master@127.13.148.61:34813 shutdown complete.
I20260926 22:49:20.686611 20465 master_runner.cc:305] Clearing existing system tablet
I20260926 22:49:20.686733 20465 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:20.686777 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:20.687433 20465 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:20.687886 20564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.688019 20465 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:20.688071 20465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.688086 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
uuid: "6988137b78334268b85f4ee28a34a66d"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "d3d16117a61c6c07a91dd261df92ee8b"
server_key_iv: "804817af6820854fdec3e53abaa7aa65"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.688661 20465 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:20.690011 20465 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 22:49:20.690062 20465 log.cc:1196] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 22:49:20.690239 20465 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Deleting consensus metadata
I20260926 22:49:20.690340 20465 master_runner.cc:315] Copying system tablet from 127.13.148.62:40093
I20260926 22:49:20.690871 20465 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: tablet copy: Beginning tablet copy session from remote peer at address 127.13.148.62:40093
I20260926 22:49:20.696238 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.691179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.13.148.62:40093)
0926 22:49:20.691219 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.691221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.691338 (+ 117us) server_negotiation.cc:407] Connection header received
0926 22:49:20.691400 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.691401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.691410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.691456 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.691744 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.692227 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.692911 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.693109 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.693161 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.694032 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.694034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.694034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.694041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.695247 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.695542 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.695543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.695544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.695599 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.695762 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.695763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.695763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.695867 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.695921 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.696078 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.696148 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:49:20.696261 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.691142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:34794)
0926 22:49:20.691273 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.691283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.691341 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.691459 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.691460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.691622 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.691675 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.691676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.692292 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.692293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.692838 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.692840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.693164 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.693165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.693225 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.693629 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.693634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.693953 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.695320 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.695321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.695322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.695480 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.695650 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.695651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.695651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.695706 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.695918 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.695919 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.695963 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.696037 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.696149 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260926 22:49:20.696720 20021 tablet_copy_service.cc:140] P 6906a7f3477746d9bee593df5a14375d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6988137b78334268b85f4ee28a34a66d ({username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:34794)
I20260926 22:49:20.696805 20021 tablet_copy_service.cc:161] P 6906a7f3477746d9bee593df5a14375d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6988137b78334268b85f4ee28a34a66d at {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:34794: session id = 6988137b78334268b85f4ee28a34a66d-00000000000000000000000000000000
I20260926 22:49:20.697575 20021 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 22:49:20.698109 20465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 22:49:20.698936 20465 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: tablet copy: Starting download of 0 data blocks...
I20260926 22:49:20.699061 20465 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: tablet copy: Starting download of 1 WAL segments...
I20260926 22:49:20.699860 20465 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 22:49:20.700541 20021 tablet_copy_service.cc:342] P 6906a7f3477746d9bee593df5a14375d: Request end of tablet copy session 6988137b78334268b85f4ee28a34a66d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:34794
I20260926 22:49:20.700596 20021 tablet_copy_service.cc:433] P 6906a7f3477746d9bee593df5a14375d: ending tablet copy session 6988137b78334268b85f4ee28a34a66d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6988137b78334268b85f4ee28a34a66d
I20260926 22:49:20.702740 20465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:20.703760 20571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:20.703909 20574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.703945 20465 server_base.cc:1034] running on GCE node
W20260926 22:49:20.705853 20572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.705978 20465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:20.706120 20465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:20.707248 20465 hybrid_clock.cc:648] HybridClock initialized: now 1790462960707240 us; error 21 us; skew 500 ppm
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:20.708521 20465 init.cc:377] Logged in from keytab as oryx/127.13.148.61@KRBTEST.COM (short username oryx)
I20260926 22:49:20.708781 20465 webserver.cc:460] Webserver started at http://127.13.148.61:32807/ using document root <none> and password file <none>
I20260926 22:49:20.708886 20465 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:20.708920 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:20.709561 20465 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:20.710069 20581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.710201 20465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.710256 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
uuid: "6988137b78334268b85f4ee28a34a66d"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "d3d16117a61c6c07a91dd261df92ee8b"
server_key_iv: "804817af6820854fdec3e53abaa7aa65"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.710306 20465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:20.724493 20465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:20.725914 20465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:20.728792 20465 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:34813
I20260926 22:49:20.729014 20633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:34813 every 8 connection(s)
I20260926 22:49:20.729153 20465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
I20260926 22:49:20.729861 20634 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:49:20.730252 20634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Bootstrap starting.
I20260926 22:49:20.732287 20634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:49:20.732475 20634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Bootstrap complete.
I20260926 22:49:20.732654 20634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } }
I20260926 22:49:20.732718 20634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6988137b78334268b85f4ee28a34a66d, State: Initialized, Role: LEARNER
I20260926 22:49:20.732762 20634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } }
I20260926 22:49:20.732918 20634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:20.732913 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } }
I20260926 22:49:20.732987 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: This master's current role is: LEARNER
I20260926 22:49:20.734200 20648 catalog_manager.cc:1259] Loaded cluster ID: a71ea67d9f974a92ae76bc21db738a4c
I20260926 22:49:20.734237 20648 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: loading cluster ID for follower catalog manager: success
I20260926 22:49:20.735136 20648 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: acquiring CA information for follower catalog manager: success
I20260926 22:49:20.735390 20648 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:49:20.745496 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.739923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.13.148.61:34813)
0926 22:49:20.740034 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.740036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.740044 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:20.740061 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.740062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.740070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.740120 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.740608 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.741223 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.741961 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.742158 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.742212 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.743279 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.743281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.743281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.743288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.744521 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.744822 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.744823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.744823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.744873 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.745032 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.745033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.745034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.745129 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.745213 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.745316 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.745392 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
WARNING: no policy specified for oryx/127.13.148.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.60.keytab.
Entry for principal oryx/127.13.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.60.keytab.
I20260926 22:49:20.770516 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:35707
--webserver_interface=127.13.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.60.keytab
--principal=oryx/127.13.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40093,127.13.148.61:34813,127.13.148.60:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:20.855270 20652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:20.855468 20652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:20.855496 20652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:20.856598 20652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:20.856634 20652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:20.856652 20652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:20.856667 20652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:20.856683 20652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:20.858155 20652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40093,127.13.148.61:34813,127.13.148.60:35707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.60.keytab
--principal=oryx/127.13.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:20.858392 20652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:20.858588 20652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:20.863062 20657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:20.863071 20658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.863169 20652 server_base.cc:1034] running on GCE node
W20260926 22:49:20.863071 20660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:20.863500 20652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:20.863788 20652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:20.864938 20652 hybrid_clock.cc:648] HybridClock initialized: now 1790462960864918 us; error 31 us; skew 500 ppm
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:20.867235 20652 init.cc:377] Logged in from keytab as oryx/127.13.148.60@KRBTEST.COM (short username oryx)
I20260926 22:49:20.867660 20652 webserver.cc:460] Webserver started at http://127.13.148.60:41771/ using document root <none> and password file <none>
I20260926 22:49:20.867865 20652 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:20.867911 20652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:20.868036 20652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:20.869087 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data/instance:
uuid: "4da3bae19a744d45a2c6c983460ef117"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "c21dca4edec256a6ca8222554710eee7"
server_key_iv: "d031f5b5b5388724fa96da4ec8f2a10d"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.869452 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal/instance:
uuid: "4da3bae19a744d45a2c6c983460ef117"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "c21dca4edec256a6ca8222554710eee7"
server_key_iv: "d031f5b5b5388724fa96da4ec8f2a10d"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.870692 20652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:20.871320 20667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.871459 20652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:20.871544 20652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
uuid: "4da3bae19a744d45a2c6c983460ef117"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "c21dca4edec256a6ca8222554710eee7"
server_key_iv: "d031f5b5b5388724fa96da4ec8f2a10d"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.871608 20652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:20.886425 20652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:20.888352 20652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:20.888477 20652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:20.892033 20652 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:35707
I20260926 22:49:20.892063 20719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:35707 every 8 connection(s)
I20260926 22:49:20.892375 20652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
I20260926 22:49:20.893244 20720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:20.894665 20720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:49:20.895273 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 20652
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
I20260926 22:49:20.903105 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.895387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.13.148.62:40093)
0926 22:49:20.895438 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.895441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.895595 (+ 154us) server_negotiation.cc:407] Connection header received
0926 22:49:20.896204 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.896206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.896218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.896268 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.896760 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.897237 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.898228 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.898491 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.898538 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.900536 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.900537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.900538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.900544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.901918 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.902251 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.902252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.902252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.902309 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.902494 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.902495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.902495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.902617 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.902667 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.902965 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.903023 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:49:20.903260 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.895303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:34800)
0926 22:49:20.895479 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.895511 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.896110 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.896294 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.896296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.896536 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.896677 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.896681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.897317 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.897317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.898162 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.898164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.898565 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.898566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.898694 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.899162 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.899170 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.900456 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.901984 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.901990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.901991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.902184 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.902365 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.902365 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.902366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.902446 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.902672 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.902677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.902841 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.902926 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.903068 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260926 22:49:20.903679 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.895755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.13.148.60:35707)
0926 22:49:20.895879 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.895884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.895918 (+ 34us) server_negotiation.cc:407] Connection header received
0926 22:49:20.896119 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.896128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.896152 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.896240 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.896775 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.897713 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.898423 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.898775 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.898839 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.901069 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.901077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.901077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.901084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.902527 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.902852 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.902854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.902854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.902900 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.903074 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.903075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.903075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.903236 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.903371 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.903446 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.903588 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260926 22:49:20.903746 20720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260926 22:49:20.908710 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.903947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.0.0.1:33564)
0926 22:49:20.903985 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.903995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.904032 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.904183 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.904184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.904309 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.904339 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.904341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.904902 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.904902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.905354 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.905356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.905628 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.905628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.905665 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.905968 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.905972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.906900 (+ 928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.908066 (+ 1166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.908067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.908067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.908206 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.908339 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.908339 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.908340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.908378 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.908528 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.908528 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.908556 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.908597 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.908655 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 22:49:20.908739 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.904035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.13.148.61:34813)
0926 22:49:20.904076 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.904078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.904086 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:20.904110 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.904111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.904122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.904182 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.904390 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.904849 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.905406 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.905583 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.905624 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.906977 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.906978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.906978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.906985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.907998 (+ 1013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.908262 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.908263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.908263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.908306 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.908408 (+ 102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.908409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.908409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.908490 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.908530 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.908619 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.908678 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:20.909060 20720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462960, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
I20260926 22:49:20.914101 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.909266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.0.0.1:38884)
0926 22:49:20.909302 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.909315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.909347 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.909464 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.909465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.909572 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.909595 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.909596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.910226 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.910226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.910691 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.910693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.911012 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.911012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.911043 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.911369 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.911373 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.912323 (+ 950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.913397 (+ 1074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.913398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.913398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.913533 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.913662 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.913662 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.913662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.913703 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.913898 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.913898 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.913927 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.913970 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.914033 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 22:49:20.914162 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.909307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.13.148.60:35707)
0926 22:49:20.909338 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.909340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.909347 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:20.909383 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.909383 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.909390 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.909436 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.909643 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.910136 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.910745 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.910965 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.911010 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.912386 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.912387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.912388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.912395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.913340 (+ 945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.913583 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.913583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.913584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.913622 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.913772 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.913773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.913773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.913857 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.913901 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.913985 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.914073 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":15}
I20260926 22:49:20.915390 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Bootstrap starting.
I20260926 22:49:20.916131 20720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:20.916373 20720 log.cc:824] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:20.916968 20720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: No bootstrap required, opened a new log
I20260926 22:49:20.918398 20720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } }
I20260926 22:49:20.918512 20720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:20.918541 20720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da3bae19a744d45a2c6c983460ef117, State: Initialized, Role: FOLLOWER
I20260926 22:49:20.918651 20720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } }
I20260926 22:49:20.918875 20725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } } }
I20260926 22:49:20.918921 20720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:20.918968 20725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:49:20.920389 20736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:49:20.920442 20736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:49:20.922313 20652 master_runner.cc:428] Detected that this master 4da3bae19a744d45a2c6c983460ef117 is joining an existing cluster
I20260926 22:49:20.922351 20652 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.148.60:35707
I20260926 22:49:20.929322 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.923400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.13.148.62:40093)
0926 22:49:20.923438 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.923440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.923640 (+ 200us) server_negotiation.cc:407] Connection header received
0926 22:49:20.923726 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.923727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.923736 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.923762 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.924120 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.924799 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.925870 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.926130 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.926166 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.927016 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.927017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.927018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.927024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.928382 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.928654 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.928655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.928655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.928730 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.928891 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.928892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.928892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.929013 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.929080 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.929164 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.929233 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:20.929425 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.923481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:34802)
0926 22:49:20.923591 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.923599 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.923661 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.923816 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.923816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.923990 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.924042 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.924043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.924887 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.924888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.925792 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.925793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.926252 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.926253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.926311 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.926657 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.926661 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.926929 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.928453 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.928454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.928454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.928596 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.928789 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.928789 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.928789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.928845 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.929066 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.929067 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.929096 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.929138 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.929203 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":46,"thread_start_us":63,"threads_started":1}
Sep 26 22:49:20 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462959, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
I20260926 22:49:20.955147 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 20063
I20260926 22:49:20.957612 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.923168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.0.0.1:38900)
0926 22:49:20.923297 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.923306 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.923366 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.923499 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.923500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.923663 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.923742 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.923744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.924533 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.924534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.925317 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.925319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.925788 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.925789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.925839 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.926228 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.926233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.954388 (+ 28155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.956411 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.956411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.956412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.956589 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.956760 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.956760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.956761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.957033 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.957321 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.957322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.957360 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.957414 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.957508 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260926 22:49:20.957947 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.923180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.13.148.60:35707)
0926 22:49:20.923212 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.923214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.923346 (+ 132us) server_negotiation.cc:407] Connection header received
0926 22:49:20.923422 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.923423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.923430 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.923454 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.923808 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.924462 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.925392 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.925662 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.925693 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.954485 (+ 28792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.954486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.954487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.954493 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.956332 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.956658 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.956658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.956659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.956704 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.957101 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.957102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.957102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.957246 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.957301 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.957804 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.957868 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:49:20.959142 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.923366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.13.148.61:34813)
0926 22:49:20.923404 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.923406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.923501 (+ 95us) server_negotiation.cc:407] Connection header received
0926 22:49:20.923598 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.923598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.923607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.923635 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.924063 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.924778 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.925906 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.926196 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.926236 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.954836 (+ 28600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.954838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.954838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.954846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.957550 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.958351 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.958352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.958352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.958410 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.958597 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.958598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.958598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.958737 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.958786 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.958973 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.959045 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:20.959630 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.923198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.0.0.1:33572)
0926 22:49:20.923449 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.923459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.923533 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.923691 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.923692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.923916 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.923976 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.923978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.924870 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.924871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.925810 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.925812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.926304 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.926305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.926365 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.954168 (+ 27803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.954174 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.954723 (+ 549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.957624 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.957625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.957625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.958270 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.958481 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.958481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.958482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.958532 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.958816 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.958816 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.958868 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.959431 (+ 563us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.959529 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":24,"thread_start_us":68,"threads_started":1}
I20260926 22:49:20.959896 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.931045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.0.0.1:38912)
0926 22:49:20.931180 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.931189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.931237 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.931354 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.931355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.931486 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.931516 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.931518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.932112 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.932113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.932600 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.932601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.932975 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.932976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.933036 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.933368 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.933373 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.934522 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.958886 (+ 24364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.958888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.958888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.959095 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.959318 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.959319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.959319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.959378 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.959615 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.959616 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.959662 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.959709 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.959811 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260926 22:49:20.960193 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.931077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.13.148.60:35707)
0926 22:49:20.931205 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.931206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.931215 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:20.931294 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.931295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.931303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.931353 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.931583 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.932050 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.932684 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.932916 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.932967 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.934604 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.934605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.934606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.934612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.958798 (+ 24186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.959173 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.959174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.959174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.959236 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.959440 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.959440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.959441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.959545 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.959999 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.960034 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.960105 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260926 22:49:20.961030 20001 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.148.60:35707: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4da3bae19a744d45a2c6c983460ef117"
member_type: NON_VOTER
last_known_addr {
host: "127.13.148.60"
port: 35707
}
attrs {
promote: true
}
}
dest_uuid: "6906a7f3477746d9bee593df5a14375d"
cas_config_opid_index: 5
I20260926 22:49:20.961295 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:43653
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:40093
--webserver_port=38679
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813,127.13.148.60:35707 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:49:20.961292 20001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } }
I20260926 22:49:20.961722 20559 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.962118 20694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:20.962685 20559 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.148.60:35707
I20260926 22:49:20.962750 20559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.962813 20559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:20.962982 20694 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 FOLLOWER]: Refusing update from remote peer 6906a7f3477746d9bee593df5a14375d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 22:49:20.963090 20652 master.cc:537] Master@127.13.148.60:35707 shutting down...
I20260926 22:49:20.963229 20559 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 22:49:20.963868 19987 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d -> Peer 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60:35707): Couldn't send request to peer 4da3bae19a744d45a2c6c983460ef117. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 22:49:20.966362 20652 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:20.966454 20652 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:20.966496 20652 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: stopping tablet replica
I20260926 22:49:20.967306 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.962971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.13.148.61:34813)
0926 22:49:20.963023 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.963026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.963552 (+ 526us) server_negotiation.cc:407] Connection header received
0926 22:49:20.963577 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.963578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.963600 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.963658 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:20.964277 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.964971 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.966373 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.966974 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.967037 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.967110 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.967213 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.967248 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:49:20.967418 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.962880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.0.0.1:33588)
0926 22:49:20.963102 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.963113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.963155 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.964153 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.964155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.964157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:20.964200 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.964202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.965055 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.965056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.966282 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.966285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.967055 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.967057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.967125 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.967150 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.967203 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.967249 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260926 22:49:20.967653 20608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 LEARNER]: Refusing update from remote peer 6906a7f3477746d9bee593df5a14375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 22:49:20.967875 20559 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 22:49:20.968457 20608 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.969607 20637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.969713 20637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: This master's current role is: LEARNER
I20260926 22:49:20.969602 20749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6906a7f3477746d9bee593df5a14375d. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: NON_VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: true } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.969817 20749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: This master's current role is: LEARNER
I20260926 22:49:20.970446 20560 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: attempting to promote NON_VOTER 6988137b78334268b85f4ee28a34a66d to VOTER
I20260926 22:49:20.970644 20560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } }
I20260926 22:49:20.971047 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.970947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.0.0.1:38916)
0926 22:49:20.970983 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.971019 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 22:49:20.971154 20608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 LEARNER]: Refusing update from remote peer 6906a7f3477746d9bee593df5a14375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 22:49:20.971480 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.971390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.0.0.1:38922)
0926 22:49:20.971429 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.971453 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 22:49:20.971578 19987 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d -> Peer 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60:35707): Couldn't send request to peer 4da3bae19a744d45a2c6c983460ef117. Status: Network error: Client connection negotiation failed: client connection to 127.13.148.60:35707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 22:49:20.971665 20746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 22:49:20.972290 20559 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6988137b78334268b85f4ee28a34a66d (127.13.148.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.972496 20608 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6988137b78334268b85f4ee28a34a66d (127.13.148.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.973187 20559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.973254 20559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:20.973312 20559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.973359 20559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:20.973589 20749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:20.973780 20749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:49:20.979568 20652 master.cc:552] Master@127.13.148.60:35707 shutdown complete.
I20260926 22:49:20.979614 20652 master_runner.cc:305] Clearing existing system tablet
I20260926 22:49:20.979738 20652 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:20.979780 20652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:20.980432 20652 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:20.980906 20752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.981041 20652 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:20.981087 20652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:20.981107 20652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
uuid: "4da3bae19a744d45a2c6c983460ef117"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "c21dca4edec256a6ca8222554710eee7"
server_key_iv: "d031f5b5b5388724fa96da4ec8f2a10d"
server_key_version: "encryptionkey@0"
I20260926 22:49:20.981822 20652 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:20.983156 20652 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 22:49:20.983227 20652 log.cc:1196] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 22:49:20.983402 20652 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Deleting consensus metadata
I20260926 22:49:20.983505 20652 master_runner.cc:315] Copying system tablet from 127.13.148.62:40093
I20260926 22:49:20.984059 20652 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: tablet copy: Beginning tablet copy session from remote peer at address 127.13.148.62:40093
I20260926 22:49:20.990806 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.984342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:34812)
0926 22:49:20.984450 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:20.984461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:20.984517 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:20.984647 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:20.984647 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:20.984831 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:20.984878 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.984880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.985838 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.985840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.986618 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:20.986621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.987126 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.987127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:20.987184 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.987643 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:20.987649 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:20.988014 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:20.989709 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.989710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.989711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.989945 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.990149 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:20.990150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:20.990150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:20.990212 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:20.990472 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:20.990472 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:20.990529 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 22:49:20.990595 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 22:49:20.990699 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
I20260926 22:49:20.990809 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:20.984463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.13.148.62:40093)
0926 22:49:20.984505 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:20.984507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:20.984518 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:49:20.984582 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:20.984583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:20.984593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:20.984644 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:20.984951 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.985708 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.986713 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:20.987050 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:20.987124 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:20.988099 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:20.988101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:20.988101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:20.988108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:20.989628 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.990019 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.990020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.990021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.990086 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:20.990276 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:20.990276 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:20.990277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:20.990409 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:20.990476 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:20.990627 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 22:49:20.990727 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:49:20.991482 20021 tablet_copy_service.cc:140] P 6906a7f3477746d9bee593df5a14375d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4da3bae19a744d45a2c6c983460ef117 ({username='oryx', principal='oryx/127.13.148.60@KRBTEST.COM'} at 127.0.0.1:34812)
I20260926 22:49:20.991545 20021 tablet_copy_service.cc:161] P 6906a7f3477746d9bee593df5a14375d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4da3bae19a744d45a2c6c983460ef117 at {username='oryx', principal='oryx/127.13.148.60@KRBTEST.COM'} at 127.0.0.1:34812: session id = 4da3bae19a744d45a2c6c983460ef117-00000000000000000000000000000000
I20260926 22:49:20.992121 20021 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 22:49:20.992604 20652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 22:49:20.993957 20652 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: tablet copy: Starting download of 0 data blocks...
I20260926 22:49:20.994097 20652 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: tablet copy: Starting download of 1 WAL segments...
I20260926 22:49:20.994776 20652 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 22:49:20.995631 20021 tablet_copy_service.cc:342] P 6906a7f3477746d9bee593df5a14375d: Request end of tablet copy session 4da3bae19a744d45a2c6c983460ef117-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.148.60@KRBTEST.COM'} at 127.0.0.1:34812
I20260926 22:49:20.995684 20021 tablet_copy_service.cc:433] P 6906a7f3477746d9bee593df5a14375d: ending tablet copy session 4da3bae19a744d45a2c6c983460ef117-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4da3bae19a744d45a2c6c983460ef117
I20260926 22:49:20.998207 20652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:20.999231 20759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:21.002525 20760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:21.004226 20762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:21.005025 20652 server_base.cc:1034] running on GCE node
I20260926 22:49:21.005125 20652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:21.005263 20652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:21.006402 20652 hybrid_clock.cc:648] HybridClock initialized: now 1790462961006395 us; error 21 us; skew 500 ppm
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:21.007939 20652 init.cc:377] Logged in from keytab as oryx/127.13.148.60@KRBTEST.COM (short username oryx)
I20260926 22:49:21.008298 20652 webserver.cc:460] Webserver started at http://127.13.148.60:34717/ using document root <none> and password file <none>
I20260926 22:49:21.008424 20652 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:21.008460 20652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:21.009227 20652 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:21.009821 20769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.009933 20652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.009985 20652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
uuid: "4da3bae19a744d45a2c6c983460ef117"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "c21dca4edec256a6ca8222554710eee7"
server_key_iv: "d031f5b5b5388724fa96da4ec8f2a10d"
server_key_version: "encryptionkey@0"
I20260926 22:49:21.010035 20652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:21.019315 20652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:21.020946 20652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:21.023779 20652 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:35707
I20260926 22:49:21.024111 20821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:35707 every 8 connection(s)
I20260926 22:49:21.024159 20652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
I20260926 22:49:21.025019 20822 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:49:21.025509 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Bootstrap starting.
I20260926 22:49:21.028151 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:49:21.028349 20822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: Bootstrap complete.
I20260926 22:49:21.028530 20822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } }
I20260926 22:49:21.028617 20822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da3bae19a744d45a2c6c983460ef117, State: Initialized, Role: LEARNER
I20260926 22:49:21.028650 20822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } }
I20260926 22:49:21.028839 20822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:21.028813 20825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:21.028993 20825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: This master's current role is: LEARNER
I20260926 22:49:21.030251 20836 catalog_manager.cc:1259] Loaded cluster ID: a71ea67d9f974a92ae76bc21db738a4c
I20260926 22:49:21.030328 20836 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: loading cluster ID for follower catalog manager: success
I20260926 22:49:21.031546 20836 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: acquiring CA information for follower catalog manager: success
I20260926 22:49:21.031852 20836 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 22:49:21.057097 20747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:21.057308 20747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:21.057338 20747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:21.058552 20747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:21.058596 20747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:21.058630 20747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:21.060170 20747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:43653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=38679
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813,127.13.148.60:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:21.060408 20747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:21.060621 20747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:49:21.065160 20747 server_base.cc:1034] running on GCE node
W20260926 22:49:21.065160 20842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:21.065290 20844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:21.065181 20841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:21.065550 20747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:21.065807 20747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:21.066936 20747 hybrid_clock.cc:648] HybridClock initialized: now 1790462961066928 us; error 24 us; skew 500 ppm
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:21.069286 20747 init.cc:377] Logged in from keytab as oryx/127.13.148.1@KRBTEST.COM (short username oryx)
I20260926 22:49:21.069669 20747 webserver.cc:460] Webserver started at http://127.13.148.1:38679/ using document root <none> and password file <none>
I20260926 22:49:21.069901 20747 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:21.069958 20747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:21.071189 20747 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:21.071666 20851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.071803 20747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.071882 20747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "fb65cccc125342c893b4fd43be439bee"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "bcdceb4e3edd3953645d9f878efe0154"
server_key_iv: "2310800411cb7ee23293696c485d1ee4"
server_key_version: "encryptionkey@0"
I20260926 22:49:21.072178 20747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:21.080116 20747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:21.081614 20747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:21.081713 20747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:21.081915 20747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:21.082142 20747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:21.082170 20747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.082194 20747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:21.082211 20747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.087260 20747 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:43653
I20260926 22:49:21.087337 20964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:43653 every 8 connection(s)
I20260926 22:49:21.087597 20747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:21.088589 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 20747
I20260926 22:49:21.088735 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 20197
I20260926 22:49:21.094252 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:33849
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:40093
--webserver_port=36015
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813,127.13.148.60:35707 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
I20260926 22:49:21.099525 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.088428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:60999 (local address 127.13.148.61:34813)
0926 22:49:21.088484 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.088486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.088677 (+ 191us) server_negotiation.cc:407] Connection header received
0926 22:49:21.089764 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.089766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.089778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.089836 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.090878 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.091709 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.093162 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.093449 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.093485 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.096511 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.096513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.096513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.096520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.097997 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.098487 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.098488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.098488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.098545 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.098826 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.098827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.098828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.098967 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.099007 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.099350 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.099429 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:21.099617 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.088488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:44543 (local address 127.13.148.62:40093)
0926 22:49:21.088545 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.088547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.088677 (+ 130us) server_negotiation.cc:407] Connection header received
0926 22:49:21.089839 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.089840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.089851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.089896 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.090978 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.091716 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.094601 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.094871 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.094911 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.097037 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.097039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.097039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.097047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.098577 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.098896 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.098897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.098897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.098944 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.099123 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.099124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.099124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.099248 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.099289 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.099475 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.099542 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 22:49:21.100087 20969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.088242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.13.148.1:60999)
0926 22:49:21.088557 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.088600 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.089637 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.089962 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.089967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.090482 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.090779 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.090785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.091828 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.091829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.093072 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.093074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.093565 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.093565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.093698 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.094231 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.094241 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.096403 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.098077 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.098085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.098087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.098396 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.098613 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.098614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.098614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.098747 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.099049 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.099054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.099203 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.099657 (+ 454us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.099850 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":23,"thread_start_us":99,"threads_started":1}
I20260926 22:49:21.100087 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.088242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.1:44543)
0926 22:49:21.088575 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.088600 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.089782 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.089963 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.089967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.090474 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.090907 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.090908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.091830 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.091831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.093820 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.093823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.094983 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.094984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.095038 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.095441 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.095446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.096918 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.098645 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.098646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.098647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.098821 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.099009 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.099009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.099009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.099060 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.099337 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.099338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.099370 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.099683 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.099910 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":1439,"thread_start_us":97,"threads_started":1}
I20260926 22:49:21.100523 20966 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40093
I20260926 22:49:21.100559 20967 heartbeater.cc:347] Connected to a master server at 127.13.148.61:34813
I20260926 22:49:21.100626 20967 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.100629 20966 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.100814 20967 heartbeater.cc:511] Master 127.13.148.61:34813 requested a full tablet report, sending...
I20260926 22:49:21.100821 20966 heartbeater.cc:511] Master 127.13.148.62:40093 requested a full tablet report, sending...
I20260926 22:49:21.101194 20001 ts_manager.cc:194] Re-registered known tserver with Master: fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653)
I20260926 22:49:21.101279 20598 ts_manager.cc:194] Registered new tserver with Master: fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653)
I20260926 22:49:21.101675 20001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:44543
I20260926 22:49:21.102210 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.088323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:58339 (local address 127.13.148.60:35707)
0926 22:49:21.089884 (+ 1561us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.089886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.089894 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:21.089914 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.089915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.089926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.089976 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.092061 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.093014 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.094983 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.095552 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.095622 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.098343 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.098344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.098344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.098352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.100846 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.101378 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.101379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.101380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.101443 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.101633 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.101634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.101635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.101788 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.101848 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.102031 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.102117 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1537,"thread_start_us":54,"threads_started":1}
I20260926 22:49:21.102330 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.088242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.13.148.1:58339)
0926 22:49:21.088720 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.088728 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.089625 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.090018 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.090019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.090633 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.090780 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.090786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.093885 (+ 3099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.093885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.094895 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.094898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.095619 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.095620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.095684 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.096032 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.096037 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.098243 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.101010 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.101011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.101012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.101206 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.101512 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.101513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.101514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.101571 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.101875 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.101876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.101917 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.101982 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.102075 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":1,"spinlock_wait_cycles":7424,"thread_start_us":90,"threads_started":1}
I20260926 22:49:21.102677 20965 heartbeater.cc:347] Connected to a master server at 127.13.148.60:35707
I20260926 22:49:21.102726 20965 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.102795 20965 heartbeater.cc:511] Master 127.13.148.60:35707 requested a full tablet report, sending...
I20260926 22:49:21.103102 20780 ts_manager.cc:194] Registered new tserver with Master: fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653)
W20260926 22:49:21.176198 20973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:21.176393 20973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:21.176424 20973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:21.177525 20973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:21.177563 20973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:21.177596 20973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:21.179062 20973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:33849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=36015
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813,127.13.148.60:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:21.179278 20973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:21.179478 20973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:21.184062 20978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:21.184055 20979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:21.184055 20981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:21.184159 20973 server_base.cc:1034] running on GCE node
I20260926 22:49:21.184469 20973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:21.184689 20973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:21.185786 20973 hybrid_clock.cc:648] HybridClock initialized: now 1790462961185766 us; error 37 us; skew 500 ppm
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:21.188113 20973 init.cc:377] Logged in from keytab as oryx/127.13.148.2@KRBTEST.COM (short username oryx)
I20260926 22:49:21.188555 20973 webserver.cc:460] Webserver started at http://127.13.148.2:36015/ using document root <none> and password file <none>
I20260926 22:49:21.188774 20973 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:21.188836 20973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:21.190375 20973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:49:21.191012 20988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.191162 20973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:21.191248 20973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "d2d0177c27c144b9b103823eb2cab339"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "0ce50fdfb55c5b552a471c6865de71d4"
server_key_iv: "82dca725bd31c6859fdd0733d2a11870"
server_key_version: "encryptionkey@0"
I20260926 22:49:21.191574 20973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:21.201431 20973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:21.203430 20973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:21.203550 20973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:21.203759 20973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:21.204025 20973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:21.204056 20973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.204079 20973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:21.204103 20973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.209414 20973 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:33849
I20260926 22:49:21.209470 21101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:33849 every 8 connection(s)
I20260926 22:49:21.209862 20973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
I20260926 22:49:21.219904 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 20973
I20260926 22:49:21.220059 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.210607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:36055 (local address 127.13.148.62:40093)
0926 22:49:21.210651 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.210653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.210906 (+ 253us) server_negotiation.cc:407] Connection header received
0926 22:49:21.211527 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.211528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.211540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.211577 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.212433 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.213179 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.214271 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.214573 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.214634 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.217380 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.217382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.217382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.217390 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.218965 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219302 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.219303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.219304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.219359 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219530 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.219531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.219531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.219665 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.219708 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.219896 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.219974 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:21.220058 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 20331
I20260926 22:49:21.220225 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.210608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:55873 (local address 127.13.148.60:35707)
0926 22:49:21.210651 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.210653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.211029 (+ 376us) server_negotiation.cc:407] Connection header received
0926 22:49:21.211504 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.211506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.211521 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.211569 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.212413 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.213178 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.214953 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.215141 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.215191 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.217846 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.217847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.217847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.217853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.219178 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219530 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.219531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.219532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.219582 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219756 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.219757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.219758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.219858 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.219888 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.220060 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.220141 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:21.220433 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.210577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.13.148.2:55873)
0926 22:49:21.210969 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.210978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.211420 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.211906 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.211906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.212130 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.212325 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.212330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.214278 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.214278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.214870 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.214872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.215202 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.215203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.215269 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.215699 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.215703 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.217779 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.219244 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.219245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.219448 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.219635 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.219636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.219689 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.219921 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.219922 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.219952 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.220042 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.220204 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":270,"thread_start_us":50,"threads_started":1}
I20260926 22:49:21.220654 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.210577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.2:36055)
0926 22:49:21.210774 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.210819 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.211420 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.211654 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.211655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.212122 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.212327 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.212332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.213256 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.213257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.214188 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.214191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.215137 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.215138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.215203 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.215539 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.215543 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.217297 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.219035 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.219036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.219214 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.219410 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.219411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.219462 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.219753 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.219754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.219791 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.220133 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.220544 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":317,"thread_start_us":94,"threads_started":1}
I20260926 22:49:21.220992 21103 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40093
I20260926 22:49:21.221065 21103 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.221233 21103 heartbeater.cc:511] Master 127.13.148.62:40093 requested a full tablet report, sending...
I20260926 22:49:21.221562 20001 ts_manager.cc:194] Re-registered known tserver with Master: d2d0177c27c144b9b103823eb2cab339 (127.13.148.2:33849)
I20260926 22:49:21.222033 20001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:36055
I20260926 22:49:21.225327 21102 heartbeater.cc:347] Connected to a master server at 127.13.148.60:35707
I20260926 22:49:21.225374 21102 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.225441 21102 heartbeater.cc:511] Master 127.13.148.60:35707 requested a full tablet report, sending...
I20260926 22:49:21.225720 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:36819
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:40093
--webserver_port=33117
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813,127.13.148.60:35707 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:49:21.225766 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.210752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33921 (local address 127.13.148.61:34813)
0926 22:49:21.210793 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.210795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.210918 (+ 123us) server_negotiation.cc:407] Connection header received
0926 22:49:21.211526 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.211527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.211539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.211569 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.212426 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.212904 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.214276 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.214573 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.214634 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.216923 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.216924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.216925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.216931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.218580 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.218938 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.218939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.218939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.218993 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219273 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.219274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.219274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.219400 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.219461 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.225565 (+ 6104us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.225646 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:49:21.225921 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.210694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.13.148.2:33921)
0926 22:49:21.210869 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.210877 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.211413 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.211646 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.211651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.212118 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.212327 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.212332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.212996 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.212997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.214188 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.214191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.214647 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.214648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.214774 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.215387 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.215396 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.216840 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.218663 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.218669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.218671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.218868 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.219058 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.219059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.219059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.219183 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.219462 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.219468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.219615 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.225525 (+ 5910us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.225648 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":24,"thread_start_us":72,"threads_started":1}
I20260926 22:49:21.226240 21104 heartbeater.cc:347] Connected to a master server at 127.13.148.61:34813
I20260926 22:49:21.226287 21104 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.226326 21104 heartbeater.cc:511] Master 127.13.148.61:34813 requested a full tablet report, sending...
I20260926 22:49:21.226418 20780 ts_manager.cc:194] Registered new tserver with Master: d2d0177c27c144b9b103823eb2cab339 (127.13.148.2:33849)
I20260926 22:49:21.226585 20598 ts_manager.cc:194] Registered new tserver with Master: d2d0177c27c144b9b103823eb2cab339 (127.13.148.2:33849)
W20260926 22:49:21.303679 21109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:21.303864 21109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:21.303886 21109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:21.304999 21109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:21.305039 21109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:21.305075 21109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:21.306529 21109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:36819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=33117
--tserver_master_addrs=127.13.148.62:40093,127.13.148.61:34813,127.13.148.60:35707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:21.306742 21109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:21.306945 21109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:49:21.311842 21109 server_base.cc:1034] running on GCE node
W20260926 22:49:21.311811 21115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:21.311820 21114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:21.311825 21117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:21.312201 21109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:21.312422 21109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:21.313556 21109 hybrid_clock.cc:648] HybridClock initialized: now 1790462961313547 us; error 26 us; skew 500 ppm
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:21.315877 21109 init.cc:377] Logged in from keytab as oryx/127.13.148.3@KRBTEST.COM (short username oryx)
I20260926 22:49:21.316316 21109 webserver.cc:460] Webserver started at http://127.13.148.3:33117/ using document root <none> and password file <none>
I20260926 22:49:21.316507 21109 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:21.316563 21109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:21.317808 21109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:21.318300 21124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.318431 21109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.318506 21109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "53cc0cfba88643bbbfc7d77fbd24fa30"
format_stamp: "Formatted at 2026-09-26 22:49:20 on dist-test-slave-98hb"
server_key: "f50581177675f81bb988130a23268372"
server_key_iv: "fc72a5fc42b1613bf1d7cc0a78db947a"
server_key_version: "encryptionkey@0"
I20260926 22:49:21.318822 21109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:21.341473 21109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:21.343061 21109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:21.343179 21109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:21.343392 21109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:21.343657 21109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:21.343688 21109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.343710 21109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:21.343732 21109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:21.349025 21109 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36819
I20260926 22:49:21.349076 21237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36819 every 8 connection(s)
I20260926 22:49:21.349383 21109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:21.349907 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 21109
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.60@KRBTEST.COM
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
Sep 26 22:49:21 dist-test-slave-98hb krb5kdc[19956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462961, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.61@KRBTEST.COM
I20260926 22:49:21.360829 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.350655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:43723 (local address 127.13.148.60:35707)
0926 22:49:21.350696 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.350698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.351478 (+ 780us) server_negotiation.cc:407] Connection header received
0926 22:49:21.351505 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.351506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.351516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.351563 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.352112 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.352829 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354442 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.354742 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354779 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.357843 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.357845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.357845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.357851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.359339 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.359794 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.359795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.359796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.359853 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.360126 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.360127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.360128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.360252 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.360292 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.360649 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.360721 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:21.361205 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.350336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:52619 (local address 127.13.148.62:40093)
0926 22:49:21.350378 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.350380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.350389 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:21.351559 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.351560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.351573 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.351601 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.352114 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.352870 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354455 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.354758 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354792 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.358422 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.358424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.358424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.358431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.360015 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.360372 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.360373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.360373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.360427 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.360610 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.360611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.360611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.360737 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.360797 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.361058 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.361121 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:49:21.361359 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.350587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.13.148.3:43723)
0926 22:49:21.351399 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.351408 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.351465 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.351611 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.351612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.351929 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.352014 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.352021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.352944 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.352945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.354356 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.354358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.354857 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.354993 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.355592 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.355597 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.357721 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.359414 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.359421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.359423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.359694 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.359924 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.359924 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.359924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.360047 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.360337 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.360343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.360499 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.360946 (+ 447us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.361187 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":231,"thread_start_us":58,"threads_started":1}
I20260926 22:49:21.361956 21238 heartbeater.cc:347] Connected to a master server at 127.13.148.60:35707
I20260926 22:49:21.362057 21238 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.362239 21238 heartbeater.cc:511] Master 127.13.148.60:35707 requested a full tablet report, sending...
I20260926 22:49:21.362367 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.350344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:38855 (local address 127.13.148.61:34813)
0926 22:49:21.350387 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.350389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.350502 (+ 113us) server_negotiation.cc:407] Connection header received
0926 22:49:21.351233 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.351234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.351247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.351298 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.352114 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.352863 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354455 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.354748 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354785 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.359157 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.359158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.359159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.359166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.360579 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.361128 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.361130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.361130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.361188 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.361753 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.361754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.361754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.361879 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.361927 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.362208 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.362291 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:21.362651 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.350085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.13.148.3:38855)
0926 22:49:21.350439 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.350448 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.351020 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.351386 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.351390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.351750 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.352013 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.352019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.352952 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.352953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.354359 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.354361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.354858 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.354995 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.355541 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.355551 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.359056 (+ 3505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.360652 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.360652 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.360653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.360845 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.361595 (+ 750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.361596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.361596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.361659 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.361973 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.361974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.362061 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.362469 (+ 408us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.362560 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":108,"thread_start_us":81,"threads_started":1}
I20260926 22:49:21.362640 20780 ts_manager.cc:194] Registered new tserver with Master: 53cc0cfba88643bbbfc7d77fbd24fa30 (127.13.148.3:36819)
I20260926 22:49:21.362937 21240 heartbeater.cc:347] Connected to a master server at 127.13.148.61:34813
I20260926 22:49:21.361362 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.350085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.3:52619)
0926 22:49:21.350259 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.350300 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.350942 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.351663 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.351664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.351914 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:21.352013 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.352019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.352952 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.352953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.354359 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.354361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.354858 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.354859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.355190 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.355803 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:21.355807 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:21.358332 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:21.360096 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.360097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.360098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.360291 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.360494 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:21.360494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:21.360495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:21.360544 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:21.360814 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:21.360815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:21.360853 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.360931 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.361116 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":118,"thread_start_us":71,"threads_started":1}
I20260926 22:49:21.362985 21240 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.363049 21240 heartbeater.cc:511] Master 127.13.148.61:34813 requested a full tablet report, sending...
I20260926 22:49:21.363250 21239 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40093
I20260926 22:49:21.363296 21239 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:21.363307 20598 ts_manager.cc:194] Registered new tserver with Master: 53cc0cfba88643bbbfc7d77fbd24fa30 (127.13.148.3:36819)
I20260926 22:49:21.363348 21239 heartbeater.cc:511] Master 127.13.148.62:40093 requested a full tablet report, sending...
I20260926 22:49:21.363659 20001 ts_manager.cc:194] Re-registered known tserver with Master: 53cc0cfba88643bbbfc7d77fbd24fa30 (127.13.148.3:36819)
I20260926 22:49:21.364099 20001 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:52619
I20260926 22:49:21.365099 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.351040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.13.148.62:40093)
0926 22:49:21.351158 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.351160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.351562 (+ 402us) server_negotiation.cc:407] Connection header received
0926 22:49:21.351585 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.351586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.351597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.351653 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.352748 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.353433 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.357905 (+ 4472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.358424 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.358475 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.360678 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.360679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.360679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.360684 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.362801 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.364276 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.364277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.364277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.364327 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.364527 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.364528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.364529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.364660 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.364769 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.364939 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.365012 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260926 22:49:21.366045 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.350943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.13.148.60:35707)
0926 22:49:21.351407 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.351409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.353304 (+ 1895us) server_negotiation.cc:407] Connection header received
0926 22:49:21.355057 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.355058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.355069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.355121 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.355795 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.356493 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.357340 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.357681 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.357778 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.362048 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.362049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.362049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.362055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.364049 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.364800 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.364801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.364801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.364854 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.365343 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.365344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.365344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.365469 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.365555 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.365885 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.365958 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":57,"threads_started":1}
I20260926 22:49:21.366631 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.351085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.13.148.61:34813)
0926 22:49:21.351358 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.351360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.351980 (+ 620us) server_negotiation.cc:407] Connection header received
0926 22:49:21.352004 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.352005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.352016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.352065 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.354150 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.354849 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.356572 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.356866 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.356931 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.363836 (+ 6905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.363837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.363837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.363844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.365293 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.365593 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.365594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.365594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.365652 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.366085 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.366086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.366086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.366210 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.366299 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.366469 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.366545 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":57,"threads_started":1}
I20260926 22:49:21.366605 20001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:21.368199 20001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:21.375002 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.371486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33849 (local address 127.0.0.1:43314)
0926 22:49:21.371645 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.371653 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.371702 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.371886 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.371887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.371888 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:21.371932 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.371934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.373102 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.373103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.374130 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.374132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.374822 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.374822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.374882 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.374886 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.374926 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.374954 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":25,"thread_start_us":75,"threads_started":1}
I20260926 22:49:21.375051 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.371585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.13.148.2:33849)
0926 22:49:21.371750 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.371755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.371765 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:49:21.371801 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.371803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.371818 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.371886 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:21.372036 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.373000 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.374187 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.374728 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.374809 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.374892 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.374938 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.374974 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260926 22:49:21.375620 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.371530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36819 (local address 127.0.0.1:56274)
0926 22:49:21.372352 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.372361 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.372406 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.372544 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.372545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.372547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:21.372589 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.372590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.373329 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.373330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.374579 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.374581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.375401 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.375402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.375460 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.375464 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.375519 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.375552 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":73,"threads_started":1}
I20260926 22:49:21.375649 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.371814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.13.148.3:36819)
0926 22:49:21.372001 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.372006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.372404 (+ 398us) server_negotiation.cc:407] Connection header received
0926 22:49:21.372461 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.372463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.372477 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.372540 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:21.372635 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.373260 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.374650 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.375326 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.375387 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.375437 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.375544 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.375579 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260926 22:49:21.375696 21036 tablet_service.cc:1467] Processing CreateTablet for tablet 20ce20c277b2462aab1e6e48b1fc3760 (DEFAULT_TABLE table=test-table [id=92c4b2341626437ba27443797a40bb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:21.376072 21036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ce20c277b2462aab1e6e48b1fc3760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:21.376392 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.371473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43653 (local address 127.0.0.1:60562)
0926 22:49:21.371641 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.371651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.371688 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.372271 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.372272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.372273 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:21.372306 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.372308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.373475 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.373475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.374579 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.374581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.376178 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.376180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.376237 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.376240 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.376301 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.376337 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":4}
I20260926 22:49:21.376408 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.371829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.13.148.1:43653)
0926 22:49:21.372124 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.372128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.372138 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:49:21.372174 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.372177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.372191 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.372265 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:21.372375 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.373404 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.375464 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.376100 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.376169 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.376242 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.376315 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.376351 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
I20260926 22:49:21.378394 21172 tablet_service.cc:1467] Processing CreateTablet for tablet 20ce20c277b2462aab1e6e48b1fc3760 (DEFAULT_TABLE table=test-table [id=92c4b2341626437ba27443797a40bb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:21.378705 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ce20c277b2462aab1e6e48b1fc3760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:21.378933 21264 tablet_bootstrap.cc:492] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Bootstrap starting.
I20260926 22:49:21.379465 21264 tablet_bootstrap.cc:654] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:21.379719 21264 log.cc:824] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:21.380393 21264 tablet_bootstrap.cc:492] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: No bootstrap required, opened a new log
I20260926 22:49:21.380450 21264 ts_tablet_manager.cc:1397] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:21.381440 21266 tablet_bootstrap.cc:492] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: Bootstrap starting.
I20260926 22:49:21.381844 20899 tablet_service.cc:1467] Processing CreateTablet for tablet 20ce20c277b2462aab1e6e48b1fc3760 (DEFAULT_TABLE table=test-table [id=92c4b2341626437ba27443797a40bb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:21.382035 21266 tablet_bootstrap.cc:654] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:21.382153 20899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20ce20c277b2462aab1e6e48b1fc3760. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:21.382335 21266 log.cc:824] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:21.382624 21264 raft_consensus.cc:348] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.382782 21264 raft_consensus.cc:374] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:21.382824 21264 raft_consensus.cc:729] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d0177c27c144b9b103823eb2cab339, State: Initialized, Role: FOLLOWER
I20260926 22:49:21.382934 21264 consensus_queue.cc:260] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.383042 21266 tablet_bootstrap.cc:492] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: No bootstrap required, opened a new log
I20260926 22:49:21.383102 21266 ts_tablet_manager.cc:1397] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:21.383167 21264 ts_tablet_manager.cc:1428] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:49:21.383277 21103 heartbeater.cc:503] Master 127.13.148.62:40093 was elected leader, sending a full tablet report...
I20260926 22:49:21.384936 21266 raft_consensus.cc:348] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.385087 21266 raft_consensus.cc:374] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:21.385129 21266 raft_consensus.cc:729] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cc0cfba88643bbbfc7d77fbd24fa30, State: Initialized, Role: FOLLOWER
I20260926 22:49:21.385248 21266 consensus_queue.cc:260] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.385265 21269 tablet_bootstrap.cc:492] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: Bootstrap starting.
I20260926 22:49:21.385474 21266 ts_tablet_manager.cc:1428] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:49:21.385578 21239 heartbeater.cc:503] Master 127.13.148.62:40093 was elected leader, sending a full tablet report...
I20260926 22:49:21.386041 21269 tablet_bootstrap.cc:654] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:21.386318 21269 log.cc:824] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:21.386884 21269 tablet_bootstrap.cc:492] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: No bootstrap required, opened a new log
I20260926 22:49:21.386931 21269 ts_tablet_manager.cc:1397] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:49:21.388144 21269 raft_consensus.cc:348] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.388262 21269 raft_consensus.cc:374] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:21.388289 21269 raft_consensus.cc:729] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb65cccc125342c893b4fd43be439bee, State: Initialized, Role: FOLLOWER
I20260926 22:49:21.388391 21269 consensus_queue.cc:260] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.388624 21269 ts_tablet_manager.cc:1428] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 22:49:21.388651 20966 heartbeater.cc:503] Master 127.13.148.62:40093 was elected leader, sending a full tablet report...
W20260926 22:49:21.461122 21105 tablet.cc:2369] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:21.504452 21268 raft_consensus.cc:482] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:21.504588 21268 raft_consensus.cc:504] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.504932 21268 leader_election.cc:288] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653), 53cc0cfba88643bbbfc7d77fbd24fa30 (127.13.148.3:36819)
I20260926 22:49:21.507625 21263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.505264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:56515 (local address 127.13.148.1:43653)
0926 22:49:21.505302 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.505304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.505310 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:49:21.505334 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.505335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.505342 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.505397 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:21.505549 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.506051 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.506985 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.507329 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.507388 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.507396 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.507527 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.507560 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:49:21.507622 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.505163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:37039 (local address 127.13.148.3:36819)
0926 22:49:21.505210 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.505212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.505262 (+ 50us) server_negotiation.cc:407] Connection header received
0926 22:49:21.505332 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.505333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.505339 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.505373 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:21.505532 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.506050 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.506986 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.507328 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.507386 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.507394 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.507522 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.507553 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:21.507632 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.505122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36819 (local address 127.13.148.2:37039)
0926 22:49:21.505202 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.505210 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.505264 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.505400 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.505401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.505403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:21.505470 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.505471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.506118 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.506119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.506905 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.506907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.507395 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.507396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.507463 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.507466 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.507509 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.507549 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":80}
I20260926 22:49:21.507643 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.505120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43653 (local address 127.13.148.2:56515)
0926 22:49:21.505172 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.505184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.505232 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.505405 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.505406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.505407 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:21.505487 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.505488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.506121 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.506121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.506902 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.506904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.507396 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.507397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.507466 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.507470 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.507515 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.507554 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":39}
I20260926 22:49:21.508105 20919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce20c277b2462aab1e6e48b1fc3760" candidate_uuid: "d2d0177c27c144b9b103823eb2cab339" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb65cccc125342c893b4fd43be439bee" is_pre_election: true
I20260926 22:49:21.508188 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce20c277b2462aab1e6e48b1fc3760" candidate_uuid: "d2d0177c27c144b9b103823eb2cab339" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" is_pre_election: true
I20260926 22:49:21.508249 20919 raft_consensus.cc:2454] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d0177c27c144b9b103823eb2cab339 in term 0.
I20260926 22:49:21.508338 21192 raft_consensus.cc:2454] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d0177c27c144b9b103823eb2cab339 in term 0.
I20260926 22:49:21.508505 20989 leader_election.cc:302] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2d0177c27c144b9b103823eb2cab339, fb65cccc125342c893b4fd43be439bee; no voters:
I20260926 22:49:21.508625 21268 raft_consensus.cc:2781] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:21.508677 21268 raft_consensus.cc:482] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:21.508698 21268 raft_consensus.cc:3037] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:21.509441 21268 raft_consensus.cc:504] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.509686 21268 leader_election.cc:288] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [CANDIDATE]: Term 1 election: Requested vote from peers fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653), 53cc0cfba88643bbbfc7d77fbd24fa30 (127.13.148.3:36819)
I20260926 22:49:21.509766 20919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce20c277b2462aab1e6e48b1fc3760" candidate_uuid: "d2d0177c27c144b9b103823eb2cab339" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb65cccc125342c893b4fd43be439bee"
I20260926 22:49:21.509843 21192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20ce20c277b2462aab1e6e48b1fc3760" candidate_uuid: "d2d0177c27c144b9b103823eb2cab339" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30"
I20260926 22:49:21.509851 20919 raft_consensus.cc:3037] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:21.509922 21192 raft_consensus.cc:3037] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:21.510591 20919 raft_consensus.cc:2454] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d0177c27c144b9b103823eb2cab339 in term 1.
I20260926 22:49:21.510768 21192 raft_consensus.cc:2454] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d0177c27c144b9b103823eb2cab339 in term 1.
I20260926 22:49:21.510779 20989 leader_election.cc:302] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2d0177c27c144b9b103823eb2cab339, fb65cccc125342c893b4fd43be439bee; no voters:
I20260926 22:49:21.510949 21268 raft_consensus.cc:2781] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:21.511099 21268 raft_consensus.cc:686] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 1 LEADER]: Becoming Leader. State: Replica: d2d0177c27c144b9b103823eb2cab339, State: Running, Role: LEADER
I20260926 22:49:21.511276 21268 consensus_queue.cc:237] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } }
I20260926 22:49:21.512192 20001 catalog_manager.cc:5495] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d0177c27c144b9b103823eb2cab339 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "d2d0177c27c144b9b103823eb2cab339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:21.515383 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.512990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.13.148.60:35707)
0926 22:49:21.513026 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.513029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.513037 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:21.513089 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.513090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.513109 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.513170 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:21.513298 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.513931 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.514737 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.515102 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.515149 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.515203 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.515274 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.515321 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:49:21.515386 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.512947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.0.0.1:38944)
0926 22:49:21.512990 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:21.513000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:21.513037 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:21.513183 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:21.513184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:21.513186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:21.513232 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.513234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.513993 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.513994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.514668 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:21.514670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.515156 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.515157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:21.515221 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.515225 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:21.515271 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:49:21.515313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:49:21.515760 20796 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.10] Dedup: 1.7->[1.8-1.10]
I20260926 22:49:21.516630 20825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6906a7f3477746d9bee593df5a14375d. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:21.516724 20825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: This master's current role is: LEARNER
I20260926 22:49:21.516965 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: NON_VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: true } } }
I20260926 22:49:21.517060 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:21.517344 20750 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: attempting to promote NON_VOTER 4da3bae19a744d45a2c6c983460ef117 to VOTER
I20260926 22:49:21.517462 20750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } }
I20260926 22:49:21.517855 20608 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Refusing update from remote peer 6906a7f3477746d9bee593df5a14375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 22:49:21.518047 20746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 22:49:21.518280 20796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 LEARNER]: Refusing update from remote peer 6906a7f3477746d9bee593df5a14375d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 22:49:21.518445 20746 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 22:49:21.518682 20750 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:21.518898 20608 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:21.519488 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:21.519553 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:21.519599 20750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:21.519639 20750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:21.519790 21282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:21.519867 21282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:49:21.520078 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.517167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.13.148.2:33849)
0926 22:49:21.517201 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.517203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.517210 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:21.517238 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.517239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.517255 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.517310 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:21.517443 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.518171 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.519135 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.519387 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.519450 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.519628 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:21.519748 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:21.519828 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.519996 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.520015 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:49:21.520208 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:21.520296 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:21.520915 20796 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:21.521667 21192 raft_consensus.cc:1261] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 FOLLOWER]: Refusing update from remote peer d2d0177c27c144b9b103823eb2cab339: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:21.521719 20919 raft_consensus.cc:1261] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 1 FOLLOWER]: Refusing update from remote peer d2d0177c27c144b9b103823eb2cab339: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:21.521693 20825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6906a7f3477746d9bee593df5a14375d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:21.521809 20825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:49:21.522002 21268 consensus_queue.cc:1035] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:21.522169 21268 consensus_queue.cc:1035] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:21.527436 20000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:21.527504 20000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:21.528950 20000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6906a7f3477746d9bee593df5a14375d: Sending DeleteTablet for 3 replicas of tablet 20ce20c277b2462aab1e6e48b1fc3760
I20260926 22:49:21.529255 20899 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce20c277b2462aab1e6e48b1fc3760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:21 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:60562
I20260926 22:49:21.529304 21172 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce20c277b2462aab1e6e48b1fc3760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:21 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:56274
I20260926 22:49:21.529340 21036 tablet_service.cc:1514] Processing DeleteTablet for tablet 20ce20c277b2462aab1e6e48b1fc3760 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:21 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43314
I20260926 22:49:21.529532 21294 tablet_replica.cc:330] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: stopping tablet replica
I20260926 22:49:21.529625 21294 raft_consensus.cc:2229] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:21.529793 21294 raft_consensus.cc:2258] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:21.530292 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 19969
I20260926 22:49:21.530297 21293 tablet_replica.cc:330] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: stopping tablet replica
I20260926 22:49:21.530407 21294 ts_tablet_manager.cc:1905] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:21.530922 21293 raft_consensus.cc:2229] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:21.531023 21293 raft_consensus.cc:2258] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:21.531435 21293 ts_tablet_manager.cc:1905] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:21.533087 21293 ts_tablet_manager.cc:1918] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:21.533162 21293 log.cc:1196] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/20ce20c277b2462aab1e6e48b1fc3760
I20260926 22:49:21.533433 21293 ts_tablet_manager.cc:1939] T 20ce20c277b2462aab1e6e48b1fc3760 P 53cc0cfba88643bbbfc7d77fbd24fa30: Deleting consensus metadata
I20260926 22:49:21.534232 21294 ts_tablet_manager.cc:1918] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:21.534278 21294 log.cc:1196] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/20ce20c277b2462aab1e6e48b1fc3760
I20260926 22:49:21.534495 21294 ts_tablet_manager.cc:1939] T 20ce20c277b2462aab1e6e48b1fc3760 P d2d0177c27c144b9b103823eb2cab339: Deleting consensus metadata
W20260926 22:49:21.536360 21126 connection.cc:535] server connection from 127.0.0.1:56274 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 22:49:21.536633 20990 connection.cc:535] server connection from 127.0.0.1:43314 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 22:49:21.537037 21292 tablet_replica.cc:330] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: stopping tablet replica
I20260926 22:49:21.537138 21292 raft_consensus.cc:2229] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:21.537258 21292 raft_consensus.cc:2258] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:21.537555 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.530002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.13.148.60:35707)
0926 22:49:21.530036 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.530038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.530071 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:49:21.530174 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.530175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.530185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.530214 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.531166 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.531885 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.532847 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.533131 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.533168 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.534732 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.534733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.534733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.534740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.536280 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.536674 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.536675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.536676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.536745 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.536964 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.536965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.536965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.537091 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.537161 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.537388 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.537469 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:49:21.537714 21292 ts_tablet_manager.cc:1905] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:21.539179 21292 ts_tablet_manager.cc:1918] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:21.539259 21292 log.cc:1196] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/20ce20c277b2462aab1e6e48b1fc3760
I20260926 22:49:21.539462 21292 ts_tablet_manager.cc:1939] T 20ce20c277b2462aab1e6e48b1fc3760 P fb65cccc125342c893b4fd43be439bee: Deleting consensus metadata
W20260926 22:49:21.539755 20852 connection.cc:441] server connection from 127.0.0.1:60562 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60562 (request call id 1) could send its response
I20260926 22:49:21.544802 21249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.538364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.13.148.60:35707)
0926 22:49:21.538395 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.538396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.538478 (+ 82us) server_negotiation.cc:407] Connection header received
0926 22:49:21.538526 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.538527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.538535 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.538577 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.539057 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.539771 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.540878 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.541074 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.541124 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.542484 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.542485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.542486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.542492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.543743 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.544037 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.544038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.544038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.544090 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.544276 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.544277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.544278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.544395 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.544454 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.544651 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.544716 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:49:21.545151 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:21.538122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.13.148.61:34813)
0926 22:49:21.538163 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:21.538165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:21.538244 (+ 79us) server_negotiation.cc:407] Connection header received
0926 22:49:21.538329 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:21.538330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:21.538342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:21.538368 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:21.538992 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.539763 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.540861 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:21.541167 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:21.541245 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:21.542484 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:21.542485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:21.542486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:21.542493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:21.544034 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.544348 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.544348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.544349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.544413 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:21.544589 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:21.544590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:21.544590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:21.544710 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:21.544798 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:21.544981 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 22:49:21.545061 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:22.525429 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:22.525176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.1:49589)
0926 22:49:22.525292 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:22.525371 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260926 22:49:22.525722 21303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:22.525456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.2:39271)
0926 22:49:22.525609 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:22.525678 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":44,"threads_started":1}
W20260926 22:49:22.525944 21103 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:40093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
W20260926 22:49:22.526083 20966 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:40093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
I20260926 22:49:22.526674 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:22.526426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.3:47235)
0926 22:49:22.526576 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:22.526634 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":43,"threads_started":1}
W20260926 22:49:22.526850 21239 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:40093 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
I20260926 22:49:23.028963 21305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6906a7f3477746d9bee593df5a14375d)
I20260926 22:49:23.029102 21305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } }
I20260926 22:49:23.029145 21306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6906a7f3477746d9bee593df5a14375d)
I20260926 22:49:23.029209 21306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } }
I20260926 22:49:23.029417 21306 leader_election.cc:288] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6906a7f3477746d9bee593df5a14375d (127.13.148.62:40093), 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60:35707)
I20260926 22:49:23.029417 21305 leader_election.cc:288] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6906a7f3477746d9bee593df5a14375d (127.13.148.62:40093), 6988137b78334268b85f4ee28a34a66d (127.13.148.61:34813)
I20260926 22:49:23.029846 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.029512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:35258)
0926 22:49:23.029683 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.029806 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":59,"threads_started":1}
I20260926 22:49:23.029999 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.029671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:35272)
0926 22:49:23.029811 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.029969 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":45,"threads_started":1}
W20260926 22:49:23.029987 20773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
W20260926 22:49:23.030200 20585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
I20260926 22:49:23.030434 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.030326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:35276)
0926 22:49:23.030385 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.030409 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 22:49:23.030504 20773 leader_election.cc:334] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6906a7f3477746d9bee593df5a14375d (127.13.148.62:40093): Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
I20260926 22:49:23.030581 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.030502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:35278)
0926 22:49:23.030538 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.030558 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 22:49:23.030647 20585 leader_election.cc:334] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6906a7f3477746d9bee593df5a14375d (127.13.148.62:40093): Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
I20260926 22:49:23.032449 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.029677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.13.148.60:35707)
0926 22:49:23.029768 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:23.029770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:23.029915 (+ 145us) server_negotiation.cc:407] Connection header received
0926 22:49:23.029941 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:23.029942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:23.029952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:23.030041 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:23.030212 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.030878 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.031773 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.032127 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.032184 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.032194 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:23.032343 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 22:49:23.032381 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":76,"thread_start_us":37,"threads_started":1}
I20260926 22:49:23.032440 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.029580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:35707 (local address 127.0.0.1:38956)
0926 22:49:23.029790 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.029801 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:23.029861 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:23.030059 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:23.030060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:23.030063 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:23.030123 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.030125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.030949 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.030950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.031694 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.031696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.032193 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.032194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.032284 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.032288 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:23.032332 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:49:23.032376 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":39,"threads_started":1}
I20260926 22:49:23.032636 20796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6988137b78334268b85f4ee28a34a66d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4da3bae19a744d45a2c6c983460ef117" is_pre_election: true
I20260926 22:49:23.032732 20796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6988137b78334268b85f4ee28a34a66d in term 1.
I20260926 22:49:23.032882 20584 leader_election.cc:302] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da3bae19a744d45a2c6c983460ef117, 6988137b78334268b85f4ee28a34a66d; no voters: 6906a7f3477746d9bee593df5a14375d
I20260926 22:49:23.032991 21307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.029551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:34813 (local address 127.0.0.1:33628)
0926 22:49:23.029862 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.029871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:23.030214 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:23.030756 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:23.030757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:23.030760 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:23.030796 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.030797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.031384 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.031385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.032126 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.032128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.032763 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.032764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.032826 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.032830 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:23.032885 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 22:49:23.032929 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":50,"threads_started":1}
I20260926 22:49:23.032994 21306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 22:49:23.032996 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.029758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.13.148.61:34813)
0926 22:49:23.029824 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:23.029826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:23.030188 (+ 362us) server_negotiation.cc:407] Connection header received
0926 22:49:23.030650 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:23.030651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:23.030661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:23.030718 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:23.030857 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.031317 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.032223 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.032673 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.032732 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.032742 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:23.032902 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 22:49:23.032939 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52,"thread_start_us":30,"threads_started":1}
I20260926 22:49:23.033032 21306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6906a7f3477746d9bee593df5a14375d)
I20260926 22:49:23.033079 21306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:49:23.033301 20608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4da3bae19a744d45a2c6c983460ef117" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6988137b78334268b85f4ee28a34a66d" is_pre_election: true
I20260926 22:49:23.033936 21306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } }
I20260926 22:49:23.034060 20608 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4da3bae19a744d45a2c6c983460ef117 in current term 2: Already voted for candidate 6988137b78334268b85f4ee28a34a66d in this term.
I20260926 22:49:23.034066 21306 leader_election.cc:288] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [CANDIDATE]: Term 2 election: Requested vote from peers 6906a7f3477746d9bee593df5a14375d (127.13.148.62:40093), 4da3bae19a744d45a2c6c983460ef117 (127.13.148.60:35707)
I20260926 22:49:23.034222 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.034137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:35288)
0926 22:49:23.034170 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.034202 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 22:49:23.034206 20796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6988137b78334268b85f4ee28a34a66d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4da3bae19a744d45a2c6c983460ef117"
I20260926 22:49:23.034235 20771 leader_election.cc:302] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da3bae19a744d45a2c6c983460ef117; no voters: 6906a7f3477746d9bee593df5a14375d, 6988137b78334268b85f4ee28a34a66d
I20260926 22:49:23.034276 20796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:49:23.034543 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.034468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:35298)
0926 22:49:23.034496 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.034522 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 22:49:23.034611 20585 leader_election.cc:334] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6906a7f3477746d9bee593df5a14375d (127.13.148.62:40093): Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
I20260926 22:49:23.035019 20796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6988137b78334268b85f4ee28a34a66d in term 2.
I20260926 22:49:23.035089 21305 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:49:23.035158 20584 leader_election.cc:302] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4da3bae19a744d45a2c6c983460ef117, 6988137b78334268b85f4ee28a34a66d; no voters: 6906a7f3477746d9bee593df5a14375d
I20260926 22:49:23.035233 21306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:49:23.035377 21306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [term 2 LEADER]: Becoming Leader. State: Replica: 6988137b78334268b85f4ee28a34a66d, State: Running, Role: LEADER
I20260926 22:49:23.035452 21306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } }
I20260926 22:49:23.035756 21315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6988137b78334268b85f4ee28a34a66d. Latest consensus state: current_term: 2 leader_uuid: "6988137b78334268b85f4ee28a34a66d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:23.035843 21315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:23.035998 21317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:23.036476 21317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:23.036537 21317 catalog_manager.cc:1259] Loaded cluster ID: a71ea67d9f974a92ae76bc21db738a4c
I20260926 22:49:23.036566 21317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:23.036722 21317 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:23.036795 21317 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Loaded TSK: 0
I20260926 22:49:23.036935 21317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:49:23.070659 20597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:23.072201 20597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:23.073124 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.073027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:35312)
0926 22:49:23.073072 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.073101 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 22:49:23.073148 20796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [term 2 FOLLOWER]: Refusing update from remote peer 6988137b78334268b85f4ee28a34a66d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 22:49:23.073370 21315 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 22:49:23.073513 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.073411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.0.0.1:35314)
0926 22:49:23.073460 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.073490 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 22:49:23.073630 20585 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d -> Peer 6906a7f3477746d9bee593df5a14375d (127.13.148.62:40093): Couldn't send request to peer 6906a7f3477746d9bee593df5a14375d. Status: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 22:49:23.073971 21305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6988137b78334268b85f4ee28a34a66d. Latest consensus state: current_term: 2 leader_uuid: "6988137b78334268b85f4ee28a34a66d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:23.074087 21305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:49:23.074328 21318 mvcc.cc:205] Tried to move back new op lower bound from 7333736296745754624 to 7333736296593682432. Current Snapshot: MvccSnapshot[applied={T|T < 7333736296745754624}]
I20260926 22:49:23.074420 21306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6988137b78334268b85f4ee28a34a66d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:23.074507 21306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:23.074599 21305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6988137b78334268b85f4ee28a34a66d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6906a7f3477746d9bee593df5a14375d" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40093 } } peers { permanent_uuid: "6988137b78334268b85f4ee28a34a66d" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 34813 } attrs { promote: false } } peers { permanent_uuid: "4da3bae19a744d45a2c6c983460ef117" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 35707 } attrs { promote: false } } }
I20260926 22:49:23.074677 21305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da3bae19a744d45a2c6c983460ef117 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:49:23.079258 21311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.076175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36819 (local address 127.0.0.1:56280)
0926 22:49:23.076279 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.076288 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:23.076329 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:23.076588 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:23.076589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:23.076591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:23.076637 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.076638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.077572 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.077573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.078333 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.078335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.079039 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.079040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.079100 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.079104 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:23.079154 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:49:23.079196 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 22:49:23.079376 21326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.076175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43653 (local address 127.0.0.1:60564)
0926 22:49:23.076429 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.076438 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:23.076509 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:23.076635 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:23.076636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:23.076637 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:23.076666 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.076667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.077346 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.077347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.078445 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.078447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.079134 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.079135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.079198 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.079202 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:23.079256 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 22:49:23.079301 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":149,"thread_start_us":65,"threads_started":1}
I20260926 22:49:23.079452 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.076254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.13.148.1:43653)
0926 22:49:23.076355 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:23.076357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:23.076486 (+ 129us) server_negotiation.cc:407] Connection header received
0926 22:49:23.076570 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:23.076571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:23.076581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:23.076637 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:23.076747 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.077278 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.078523 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.079078 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.079133 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.079145 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:23.079265 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 22:49:23.079305 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":55,"threads_started":1}
I20260926 22:49:23.079627 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.076105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33849 (local address 127.0.0.1:43336)
0926 22:49:23.076139 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.076148 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:23.076181 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:23.076500 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:23.076501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:23.076502 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:23.076546 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.076548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.077549 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.077550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.078598 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.078601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.079237 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.079238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.079288 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.079292 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:23.079336 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:49:23.079569 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":1}
I20260926 22:49:23.079636 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.076281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.13.148.3:36819)
0926 22:49:23.076457 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:23.076459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:23.076467 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:23.076492 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:23.076493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:23.076504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:23.076588 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:23.076707 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.077473 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.078411 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.078961 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.079032 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.079044 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:23.079470 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 22:49:23.079567 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":61,"threads_started":1}
I20260926 22:49:23.079648 20899 tablet_service.cc:1467] Processing CreateTablet for tablet 30f614615f7f4f3bbac02a11eb47e620 (DEFAULT_TABLE table=test-table [id=c30862458e6a46b290e485c9fbd1723a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:23.079777 20899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f614615f7f4f3bbac02a11eb47e620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:23.079828 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.076186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.13.148.2:33849)
0926 22:49:23.076400 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:23.076402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:23.076408 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:49:23.076431 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:23.076431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:23.076439 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:23.076488 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:23.076736 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.077471 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.078676 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.079138 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.079191 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.079202 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:23.079721 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 22:49:23.079765 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":60,"threads_started":1}
I20260926 22:49:23.079977 21036 tablet_service.cc:1467] Processing CreateTablet for tablet 30f614615f7f4f3bbac02a11eb47e620 (DEFAULT_TABLE table=test-table [id=c30862458e6a46b290e485c9fbd1723a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:23.080041 21172 tablet_service.cc:1467] Processing CreateTablet for tablet 30f614615f7f4f3bbac02a11eb47e620 (DEFAULT_TABLE table=test-table [id=c30862458e6a46b290e485c9fbd1723a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:23.080080 21036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f614615f7f4f3bbac02a11eb47e620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:23.080150 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f614615f7f4f3bbac02a11eb47e620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:23.081218 21327 tablet_bootstrap.cc:492] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: Bootstrap starting.
I20260926 22:49:23.081229 21328 tablet_bootstrap.cc:492] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee: Bootstrap starting.
I20260926 22:49:23.081481 21329 tablet_bootstrap.cc:492] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: Bootstrap starting.
I20260926 22:49:23.081594 21327 tablet_bootstrap.cc:654] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:23.081790 21328 tablet_bootstrap.cc:654] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:23.081794 21329 tablet_bootstrap.cc:654] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:23.082091 21327 tablet_bootstrap.cc:492] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: No bootstrap required, opened a new log
I20260926 22:49:23.082145 21327 ts_tablet_manager.cc:1397] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:23.082255 21327 raft_consensus.cc:348] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.082296 21328 tablet_bootstrap.cc:492] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee: No bootstrap required, opened a new log
I20260926 22:49:23.082329 21327 raft_consensus.cc:374] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:23.082341 21328 ts_tablet_manager.cc:1397] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 22:49:23.082347 21327 raft_consensus.cc:729] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d0177c27c144b9b103823eb2cab339, State: Initialized, Role: FOLLOWER
I20260926 22:49:23.082417 21327 consensus_queue.cc:260] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.082456 21328 raft_consensus.cc:348] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.082520 21328 raft_consensus.cc:374] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:23.082548 21328 raft_consensus.cc:729] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb65cccc125342c893b4fd43be439bee, State: Initialized, Role: FOLLOWER
I20260926 22:49:23.082552 21327 ts_tablet_manager.cc:1428] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:23.082612 21328 consensus_queue.cc:260] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.082752 21328 ts_tablet_manager.cc:1428] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:23.083014 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.082784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.2:47139)
0926 22:49:23.082915 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.082949 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260926 22:49:23.083108 21104 heartbeater.cc:503] Master 127.13.148.61:34813 was elected leader, sending a full tablet report...
I20260926 22:49:23.083482 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.083278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.1:33925)
0926 22:49:23.083402 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.083441 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":85,"threads_started":1}
I20260926 22:49:23.083666 21329 tablet_bootstrap.cc:492] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: No bootstrap required, opened a new log
I20260926 22:49:23.083712 21329 ts_tablet_manager.cc:1397] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:49:23.083830 21329 raft_consensus.cc:348] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.083897 21329 raft_consensus.cc:374] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:23.083920 21329 raft_consensus.cc:729] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53cc0cfba88643bbbfc7d77fbd24fa30, State: Initialized, Role: FOLLOWER
I20260926 22:49:23.083954 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.083192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.2:50631)
0926 22:49:23.083912 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.083935 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":703}
I20260926 22:49:23.083958 21329 consensus_queue.cc:260] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.084090 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.083869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.1:35441)
0926 22:49:23.083972 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.084065 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 22:49:23.084095 21329 ts_tablet_manager.cc:1428] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:23.084456 21240 heartbeater.cc:503] Master 127.13.148.61:34813 was elected leader, sending a full tablet report...
I20260926 22:49:23.084569 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.084279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.3:54473)
0926 22:49:23.084456 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.084490 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260926 22:49:23.084570 20967 heartbeater.cc:503] Master 127.13.148.61:34813 was elected leader, sending a full tablet report...
W20260926 22:49:23.095155 20968 tablet.cc:2369] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:49:23.100841 21241 tablet.cc:2369] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:23.257316 21337 raft_consensus.cc:482] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:23.257407 21337 raft_consensus.cc:504] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.257752 21337 leader_election.cc:288] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d0177c27c144b9b103823eb2cab339 (127.13.148.2:33849), fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653)
I20260926 22:49:23.260416 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.257973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:34259 (local address 127.13.148.1:43653)
0926 22:49:23.258010 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:23.258012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:23.258197 (+ 185us) server_negotiation.cc:407] Connection header received
0926 22:49:23.258246 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:23.258247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:23.258257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:23.258302 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:23.258432 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.258884 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.259758 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.260097 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.260148 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.260157 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:23.260309 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 22:49:23.260346 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:23.260432 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.257902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:33849 (local address 127.13.148.3:53461)
0926 22:49:23.257944 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.257955 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:23.258000 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:23.258135 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:23.258136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:23.258138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:23.258193 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.258195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.258869 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.258869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.259665 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.259668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.260199 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.260200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.260264 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.260267 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:23.260317 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:49:23.260352 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 22:49:23.260432 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.257968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:53461 (local address 127.13.148.2:33849)
0926 22:49:23.258010 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:23.258012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:23.258020 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:23.258050 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:23.258051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:23.258060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:23.258127 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:23.258259 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.258795 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.259767 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.260112 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.260175 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.260185 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:23.260334 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 22:49:23.260371 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:49:23.260447 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.257902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43653 (local address 127.13.148.3:34259)
0926 22:49:23.258121 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.258131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:23.258187 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:23.258316 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:23.258317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:23.258318 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:23.258377 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.258378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.258954 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.258954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.259671 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:23.259673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.260168 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.260169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:23.260237 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.260241 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:23.260286 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:49:23.260322 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":42,"thread_start_us":108,"threads_started":1}
I20260926 22:49:23.260648 20919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f614615f7f4f3bbac02a11eb47e620" candidate_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb65cccc125342c893b4fd43be439bee" is_pre_election: true
I20260926 22:49:23.260637 21056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f614615f7f4f3bbac02a11eb47e620" candidate_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d0177c27c144b9b103823eb2cab339" is_pre_election: true
I20260926 22:49:23.260730 20919 raft_consensus.cc:2454] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53cc0cfba88643bbbfc7d77fbd24fa30 in term 0.
I20260926 22:49:23.260766 21056 raft_consensus.cc:2454] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53cc0cfba88643bbbfc7d77fbd24fa30 in term 0.
I20260926 22:49:23.260912 21125 leader_election.cc:302] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53cc0cfba88643bbbfc7d77fbd24fa30, fb65cccc125342c893b4fd43be439bee; no voters:
I20260926 22:49:23.261035 21337 raft_consensus.cc:2781] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:23.261075 21337 raft_consensus.cc:482] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:23.261093 21337 raft_consensus.cc:3037] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:23.261796 21337 raft_consensus.cc:504] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.261929 21337 leader_election.cc:288] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [CANDIDATE]: Term 1 election: Requested vote from peers d2d0177c27c144b9b103823eb2cab339 (127.13.148.2:33849), fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653)
I20260926 22:49:23.262073 20919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f614615f7f4f3bbac02a11eb47e620" candidate_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb65cccc125342c893b4fd43be439bee"
I20260926 22:49:23.262094 21056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f614615f7f4f3bbac02a11eb47e620" candidate_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d0177c27c144b9b103823eb2cab339"
I20260926 22:49:23.262145 20919 raft_consensus.cc:3037] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:23.262156 21056 raft_consensus.cc:3037] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:23.262732 20919 raft_consensus.cc:2454] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53cc0cfba88643bbbfc7d77fbd24fa30 in term 1.
I20260926 22:49:23.262779 21056 raft_consensus.cc:2454] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53cc0cfba88643bbbfc7d77fbd24fa30 in term 1.
I20260926 22:49:23.262888 21125 leader_election.cc:302] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53cc0cfba88643bbbfc7d77fbd24fa30, fb65cccc125342c893b4fd43be439bee; no voters:
I20260926 22:49:23.262984 21337 raft_consensus.cc:2781] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:23.263118 21337 raft_consensus.cc:686] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 LEADER]: Becoming Leader. State: Replica: 53cc0cfba88643bbbfc7d77fbd24fa30, State: Running, Role: LEADER
I20260926 22:49:23.263176 21337 consensus_queue.cc:237] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } }
I20260926 22:49:23.263767 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.263621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.3:51125)
0926 22:49:23.263703 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.263732 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 22:49:23.263921 20597 catalog_manager.cc:5495] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53cc0cfba88643bbbfc7d77fbd24fa30 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53cc0cfba88643bbbfc7d77fbd24fa30" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:23.303790 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.301770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.13.148.3:36819)
0926 22:49:23.301814 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:23.301816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:23.301846 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:49:23.301882 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:23.301883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:23.301900 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:23.301950 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:23.302056 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.302530 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.303205 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:23.303398 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:23.303444 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:23.303542 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:23.303637 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:23.303684 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:23.303712 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 22:49:23.303730 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:23.304819 20919 raft_consensus.cc:1261] T 30f614615f7f4f3bbac02a11eb47e620 P fb65cccc125342c893b4fd43be439bee [term 1 FOLLOWER]: Refusing update from remote peer 53cc0cfba88643bbbfc7d77fbd24fa30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:23.304889 21056 raft_consensus.cc:1261] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 1 FOLLOWER]: Refusing update from remote peer 53cc0cfba88643bbbfc7d77fbd24fa30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:23.305078 21340 consensus_queue.cc:1035] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb65cccc125342c893b4fd43be439bee" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 43653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:23.305229 21337 consensus_queue.cc:1035] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d0177c27c144b9b103823eb2cab339" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:23.305984 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.305881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.2:56877)
0926 22:49:23.305923 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.305952 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 22:49:23.306286 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.306139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.1:50373)
0926 22:49:23.306212 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.306254 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 22:49:23.307363 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.307272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.3:55097)
0926 22:49:23.307306 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.307333 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 22:49:23.307570 21343 mvcc.cc:205] Tried to move back new op lower bound from 7333736297694515200 to 7333736297526435840. Current Snapshot: MvccSnapshot[applied={T|T < 7333736297694515200 or (T in {7333736297694515200})}]
I20260926 22:49:23.307794 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.307712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.1:44433)
0926 22:49:23.307749 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.307773 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 22:49:23.308205 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.308124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.3:54893)
0926 22:49:23.308154 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.308180 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 22:49:23.309036 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.308926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40093 (local address 127.13.148.2:39799)
0926 22:49:23.308977 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.309009 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 22:49:23.309405 20597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:23.309491 20597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:23.310799 20597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6988137b78334268b85f4ee28a34a66d: Sending DeleteTablet for 3 replicas of tablet 30f614615f7f4f3bbac02a11eb47e620
I20260926 22:49:23.311082 20899 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f614615f7f4f3bbac02a11eb47e620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:23 UTC) from {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:60564
I20260926 22:49:23.311268 21036 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f614615f7f4f3bbac02a11eb47e620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:23 UTC) from {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:43336
I20260926 22:49:23.311383 21172 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f614615f7f4f3bbac02a11eb47e620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:23 UTC) from {username='oryx', principal='oryx/127.13.148.61@KRBTEST.COM'} at 127.0.0.1:56280
I20260926 22:49:23.311501 21354 tablet_replica.cc:330] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: stopping tablet replica
I20260926 22:49:23.311579 21354 raft_consensus.cc:2229] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:23.311625 21354 raft_consensus.cc:2258] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:23.311717 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 20747
I20260926 22:49:23.312654 21354 ts_tablet_manager.cc:1905] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:23.314321 21354 ts_tablet_manager.cc:1918] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:23.314379 21354 log.cc:1196] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/30f614615f7f4f3bbac02a11eb47e620
I20260926 22:49:23.314469 21355 tablet_replica.cc:330] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: stopping tablet replica
I20260926 22:49:23.314566 21355 raft_consensus.cc:2229] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:23.314605 21354 ts_tablet_manager.cc:1939] T 30f614615f7f4f3bbac02a11eb47e620 P d2d0177c27c144b9b103823eb2cab339: Deleting consensus metadata
I20260926 22:49:23.314714 21355 raft_consensus.cc:2258] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:23.314905 20585 catalog_manager.cc:4841] TS d2d0177c27c144b9b103823eb2cab339 (127.13.148.2:33849): tablet 30f614615f7f4f3bbac02a11eb47e620 (table test-table [id=c30862458e6a46b290e485c9fbd1723a]) successfully deleted
I20260926 22:49:23.315014 21355 ts_tablet_manager.cc:1905] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:23.316498 21355 ts_tablet_manager.cc:1918] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:23.316560 21355 log.cc:1196] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/30f614615f7f4f3bbac02a11eb47e620
I20260926 22:49:23.316774 21355 ts_tablet_manager.cc:1939] T 30f614615f7f4f3bbac02a11eb47e620 P 53cc0cfba88643bbbfc7d77fbd24fa30: Deleting consensus metadata
I20260926 22:49:23.317066 20583 catalog_manager.cc:4841] TS 53cc0cfba88643bbbfc7d77fbd24fa30 (127.13.148.3:36819): tablet 30f614615f7f4f3bbac02a11eb47e620 (table test-table [id=c30862458e6a46b290e485c9fbd1723a]) successfully deleted
I20260926 22:49:23.324857 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 20973
I20260926 22:49:23.324921 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:23.324791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:43653 (local address 127.0.0.1:60566)
0926 22:49:23.324845 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:23.324885 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:43653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 22:49:23.325641 20582 catalog_manager.cc:4584] TS fb65cccc125342c893b4fd43be439bee (127.13.148.1:43653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30f614615f7f4f3bbac02a11eb47e620: Network error: Client connection negotiation failed: client connection to 127.13.148.1:43653: connect: Connection refused (error 111)
I20260926 22:49:23.329984 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 21109
I20260926 22:49:23.334199 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 20465
I20260926 22:49:23.338508 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 20652
2026-09-26T22:49:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4523 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:23 dist-test-slave-98hb krb5kdc[21358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:23 dist-test-slave-98hb krb5kdc[21358](info): set up 2 sockets
Sep 26 22:49:23 dist-test-slave-98hb krb5kdc[21358](info): commencing operation
krb5kdc: starting...
W20260926 22:49:24.290171 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.62.keytab.
Entry for principal oryx/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.62.keytab.
I20260926 22:49:24.396252 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36411
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.62.keytab
--principal=oryx/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:24.472209 21371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:24.472419 21371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:24.472447 21371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:24.473568 21371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:24.473608 21371 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 22:49:24.473623 21371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:24.473639 21371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:24.473654 21371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:24.473670 21371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:24.475152 21371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36411
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.62.keytab
--principal=oryx/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:24.475386 21371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:24.475580 21371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:24.480023 21376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:24.480086 21379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:24.480057 21377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:24.480305 21371 server_base.cc:1034] running on GCE node
I20260926 22:49:24.480531 21371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:24.480800 21371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:24.481930 21371 hybrid_clock.cc:648] HybridClock initialized: now 1790462964481917 us; error 24 us; skew 500 ppm
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:24.488823 21371 init.cc:377] Logged in from keytab as oryx/127.13.148.62@KRBTEST.COM (short username oryx)
I20260926 22:49:24.489236 21371 webserver.cc:460] Webserver started at http://127.13.148.62:35043/ using document root <none> and password file <none>
I20260926 22:49:24.489437 21371 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:24.489482 21371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:24.489622 21371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:24.490702 21371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "b89c7e1748c247939ec16cddf7ab0de7"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "3b52340448de6c38cd38453fd43d20da"
server_key_iv: "ababa2db874978d1f79454a5e132ce42"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.491047 21371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "b89c7e1748c247939ec16cddf7ab0de7"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "3b52340448de6c38cd38453fd43d20da"
server_key_iv: "ababa2db874978d1f79454a5e132ce42"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.492235 21371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:49:24.492854 21386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.492990 21371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:24.493077 21371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "b89c7e1748c247939ec16cddf7ab0de7"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "3b52340448de6c38cd38453fd43d20da"
server_key_iv: "ababa2db874978d1f79454a5e132ce42"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.493134 21371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:24.515602 21371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:24.517609 21371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:24.517788 21371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:24.521889 21371 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36411
I20260926 22:49:24.521950 21448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36411 every 8 connection(s)
I20260926 22:49:24.522228 21371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:24.522795 21449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:24.524921 21449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: Bootstrap starting.
I20260926 22:49:24.525460 21449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:24.525705 21449 log.cc:824] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:24.526304 21449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: No bootstrap required, opened a new log
I20260926 22:49:24.528211 21449 raft_consensus.cc:348] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89c7e1748c247939ec16cddf7ab0de7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36411 } }
I20260926 22:49:24.528343 21449 raft_consensus.cc:374] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:24.528374 21449 raft_consensus.cc:729] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89c7e1748c247939ec16cddf7ab0de7, State: Initialized, Role: FOLLOWER
I20260926 22:49:24.528498 21449 consensus_queue.cc:260] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89c7e1748c247939ec16cddf7ab0de7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36411 } }
I20260926 22:49:24.528574 21449 raft_consensus.cc:388] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:24.528607 21449 raft_consensus.cc:482] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:24.528651 21449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:24.529121 21449 raft_consensus.cc:504] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89c7e1748c247939ec16cddf7ab0de7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36411 } }
I20260926 22:49:24.529210 21449 leader_election.cc:302] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b89c7e1748c247939ec16cddf7ab0de7; no voters:
I20260926 22:49:24.529353 21449 leader_election.cc:288] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:24.529393 21452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:24.529537 21452 raft_consensus.cc:686] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [term 1 LEADER]: Becoming Leader. State: Replica: b89c7e1748c247939ec16cddf7ab0de7, State: Running, Role: LEADER
I20260926 22:49:24.529578 21449 sys_catalog.cc:564] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:24.529625 21452 consensus_queue.cc:237] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89c7e1748c247939ec16cddf7ab0de7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36411 } }
I20260926 22:49:24.530016 21453 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b89c7e1748c247939ec16cddf7ab0de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89c7e1748c247939ec16cddf7ab0de7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36411 } } }
I20260926 22:49:24.530153 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 21371
I20260926 22:49:24.530251 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:24.530318 21453 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:24.530438 13904 external_mini_cluster.cc:1428] Setting key 11781e2e62f44612e7126f15fe170af0
I20260926 22:49:24.530408 21454 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b89c7e1748c247939ec16cddf7ab0de7. Latest consensus state: current_term: 1 leader_uuid: "b89c7e1748c247939ec16cddf7ab0de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89c7e1748c247939ec16cddf7ab0de7" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36411 } } }
I20260926 22:49:24.530473 21454 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:24.530706 21461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:24.531273 21461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:24.532606 21461 catalog_manager.cc:1347] Generated new cluster ID: 7b0dfc6008914ad687c56d397e78c654
I20260926 22:49:24.532656 21461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:24.542121 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.531078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.13.148.62:36411)
0926 22:49:24.531774 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:24.531782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:24.531805 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:49:24.531887 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:24.531892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:24.531932 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:24.532050 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:24.532801 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.534214 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.535229 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.536184 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.536301 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.538857 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:24.538861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:24.538861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:24.538872 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:24.540612 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.540982 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.540985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.540986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.541076 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.541253 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.541253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.541254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.541520 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:24.541659 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:24.541782 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 22:49:24.541911 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":71,"threads_started":1}
I20260926 22:49:24.545722 21461 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:24.546340 21461 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:24.551407 21461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: Generated new TSK 0
I20260926 22:49:24.551548 21461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
I20260926 22:49:24.575968 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36411
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:24.671665 21475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:24.671849 21475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:24.671880 21475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:24.672981 21475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:24.673019 21475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:24.673046 21475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:24.673069 21475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:24.674551 21475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:24.674763 21475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:24.674958 21475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:24.679594 21480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:24.679625 21483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:24.679730 21475 server_base.cc:1034] running on GCE node
W20260926 22:49:24.679880 21481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:24.680142 21475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:24.680408 21475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:24.681567 21475 hybrid_clock.cc:648] HybridClock initialized: now 1790462964681546 us; error 34 us; skew 500 ppm
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:24.683959 21475 init.cc:377] Logged in from keytab as oryx/127.13.148.1@KRBTEST.COM (short username oryx)
I20260926 22:49:24.684340 21475 webserver.cc:460] Webserver started at http://127.13.148.1:38159/ using document root <none> and password file <none>
I20260926 22:49:24.684542 21475 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:24.684587 21475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:24.684707 21475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:24.685704 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "42b5e765f7914ad6a7674adb8d3d96d9"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "1f472808915b63ac7e3c721433877162"
server_key_iv: "a057b41b9b5a01c4066af03458d25408"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.686079 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "42b5e765f7914ad6a7674adb8d3d96d9"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "1f472808915b63ac7e3c721433877162"
server_key_iv: "a057b41b9b5a01c4066af03458d25408"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.687220 21475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:24.687857 21490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.687999 21475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:24.688076 21475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "42b5e765f7914ad6a7674adb8d3d96d9"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "1f472808915b63ac7e3c721433877162"
server_key_iv: "a057b41b9b5a01c4066af03458d25408"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.688138 21475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:24.697880 21475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:24.699436 21475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:24.699558 21475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:24.699990 21475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:24.700028 21475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.700057 21475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:24.700073 21475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.706032 21475 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:46121
I20260926 22:49:24.706089 21605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:46121 every 8 connection(s)
I20260926 22:49:24.706386 21475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:24.709125 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.700414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48285 (local address 127.13.148.62:36411)
0926 22:49:24.700482 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:24.700485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:24.700667 (+ 182us) server_negotiation.cc:407] Connection header received
0926 22:49:24.701134 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:24.701136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:24.701153 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:24.701238 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:24.702015 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.702751 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.704209 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.704521 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.704597 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.706965 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:24.706967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:24.706967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:24.706975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:24.708177 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.708479 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.708480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.708480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.708523 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.708687 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.708687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.708687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.708775 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:24.708818 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:24.709004 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 22:49:24.709065 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:49:24.709285 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.700402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36411 (local address 127.13.148.1:48285)
0926 22:49:24.700545 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:24.700583 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:24.701040 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:24.701304 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:24.701309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:24.701629 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:24.701918 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.701925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.702869 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.702870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.704118 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.704122 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.704598 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.704599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.704691 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.705231 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:24.705241 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:24.706895 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:24.708239 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.708242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.708244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.708429 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.708566 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.708566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.708566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.708647 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.708818 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:24.708820 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:24.708920 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 22:49:24.708991 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 22:49:24.709096 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260926 22:49:24.709712 21606 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36411
I20260926 22:49:24.709828 21606 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:24.710002 21606 heartbeater.cc:511] Master 127.13.148.62:36411 requested a full tablet report, sending...
I20260926 22:49:24.710493 21402 ts_manager.cc:194] Registered new tserver with Master: 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121)
I20260926 22:49:24.710618 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 21475
I20260926 22:49:24.710728 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:24.710916 13904 external_mini_cluster.cc:1428] Setting key 356d0222bb7149865416583e19ad5b48
I20260926 22:49:24.711336 21402 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:48285
I20260926 22:49:24.716151 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.710650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:44987 (local address 127.13.148.62:36411)
0926 22:49:24.711074 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:24.711076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:24.711084 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:24.711101 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:24.711102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:24.711110 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:24.711159 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:24.711498 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.712110 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.712897 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.713125 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.713179 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.713995 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:24.713997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:24.713997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:24.714003 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:24.715203 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.715547 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.715547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.715548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.715590 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.715763 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.715763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.715763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.715845 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:24.715886 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:24.716032 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 22:49:24.716096 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260926 22:49:24.716213 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.710651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36411 (local address 127.13.148.1:44987)
0926 22:49:24.710767 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:24.710777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:24.710821 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:24.711196 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:24.711197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:24.711378 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:24.711436 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.711438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.712201 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.712202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.712789 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.712791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.713189 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.713190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.713258 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.713594 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:24.713599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:24.713929 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:24.715284 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.715285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.715286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.715482 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.715640 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.715641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.715641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.715703 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.715894 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:24.715894 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:24.715940 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 22:49:24.716018 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:49:24.716118 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
WARNING: no policy specified for oryx/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
I20260926 22:49:24.734382 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36411
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:24.812784 21615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:24.812983 21615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:24.813005 21615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:24.814174 21615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:24.814213 21615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:24.814234 21615 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:24.814265 21615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:24.815744 21615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:24.815958 21615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:24.816179 21615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:24.820732 21623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:24.820708 21621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:24.820767 21620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:24.820794 21615 server_base.cc:1034] running on GCE node
I20260926 22:49:24.821062 21615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:24.821309 21615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:24.822438 21615 hybrid_clock.cc:648] HybridClock initialized: now 1790462964822427 us; error 27 us; skew 500 ppm
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:24.824610 21615 init.cc:377] Logged in from keytab as oryx/127.13.148.2@KRBTEST.COM (short username oryx)
I20260926 22:49:24.824975 21615 webserver.cc:460] Webserver started at http://127.13.148.2:37135/ using document root <none> and password file <none>
I20260926 22:49:24.825148 21615 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:24.825194 21615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:24.825282 21615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:24.826254 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "64de6c63caf046468732d803a8926990"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "d9928b14d9e0f9c9ff3882b9f759cc97"
server_key_iv: "3b791694cfda38a2afb707c9ba606594"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.826531 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "64de6c63caf046468732d803a8926990"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "d9928b14d9e0f9c9ff3882b9f759cc97"
server_key_iv: "3b791694cfda38a2afb707c9ba606594"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.827620 21615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:24.828259 21630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.828400 21615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:24.828480 21615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "64de6c63caf046468732d803a8926990"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "d9928b14d9e0f9c9ff3882b9f759cc97"
server_key_iv: "3b791694cfda38a2afb707c9ba606594"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.828536 21615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:24.838864 21615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:24.840444 21615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:24.840569 21615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:24.841018 21615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:24.841054 21615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.841080 21615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:24.841097 21615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.847174 21615 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:46851
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:24.847506 21615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:24.847544 21745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:46851 every 8 connection(s)
I20260926 22:49:24.848074 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 21615
I20260926 22:49:24.848177 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:24.848357 13904 external_mini_cluster.cc:1428] Setting key f3b8a13ef3cad3e3d512a893dd73e6bd
I20260926 22:49:24.850767 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.841400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36411 (local address 127.13.148.2:56311)
0926 22:49:24.841539 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:24.841570 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:24.842108 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:24.842359 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:24.842364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:24.842683 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:24.842957 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.842964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.843920 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.843920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.845150 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.845152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.845580 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.845580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.845675 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.846149 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:24.846183 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:24.847788 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:24.849338 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.849344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.849345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.849601 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.849889 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.849889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.849890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.850010 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.850214 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:24.850217 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:24.850356 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 22:49:24.850464 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 22:49:24.850617 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260926 22:49:24.851059 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.841392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:56311 (local address 127.13.148.62:36411)
0926 22:49:24.841432 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:24.841434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:24.841641 (+ 207us) server_negotiation.cc:407] Connection header received
0926 22:49:24.842204 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:24.842205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:24.842219 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:24.842278 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:24.843075 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.843826 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.845230 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.845506 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.845579 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.847881 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:24.847883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:24.847883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:24.847891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:24.849270 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.849716 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.849717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.849718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.849822 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.850065 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.850066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.850066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.850165 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:24.850216 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:24.850900 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 22:49:24.850976 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:49:24.851392 21746 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36411
I20260926 22:49:24.851470 21746 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:24.851637 21746 heartbeater.cc:511] Master 127.13.148.62:36411 requested a full tablet report, sending...
I20260926 22:49:24.851992 21402 ts_manager.cc:194] Registered new tserver with Master: 64de6c63caf046468732d803a8926990 (127.13.148.2:46851)
I20260926 22:49:24.852355 21402 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:56311
WARNING: no policy specified for oryx/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260926 22:49:24.859030 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.853978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:54637 (local address 127.13.148.62:36411)
0926 22:49:24.854033 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:24.854034 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:24.854072 (+ 38us) server_negotiation.cc:407] Connection header received
0926 22:49:24.854155 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:24.854156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:24.854166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:24.854231 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:24.854479 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.855168 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.855956 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.856251 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.856324 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.857137 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:24.857138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:24.857139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:24.857145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:24.858197 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.858473 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.858474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.858474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.858517 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.858674 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.858674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.858674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.858788 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:24.858833 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:24.858923 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 22:49:24.858974 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:49:24.859081 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.853913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36411 (local address 127.13.148.2:54637)
0926 22:49:24.854021 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:24.854030 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:24.854109 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:24.854234 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:24.854235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:24.854378 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:24.854424 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.854425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.855266 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.855267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.855884 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.855886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.856341 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.856342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.856408 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.856801 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:24.856805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:24.857057 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:24.858270 (+ 1213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.858271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.858271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.858416 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.858553 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.858554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.858554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.858633 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.858831 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:24.858832 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:24.858864 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 22:49:24.858907 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:49:24.858975 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
I20260926 22:49:24.872014 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36411
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:24.949147 21755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:24.949347 21755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:24.949368 21755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:24.950459 21755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:24.950492 21755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:24.950510 21755 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 22:49:24.950526 21755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:24.951944 21755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.148.62:36411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:24.952134 21755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:24.952332 21755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:24.956837 21763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:24.956856 21760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:24.956920 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:24.957083 21755 server_base.cc:1034] running on GCE node
I20260926 22:49:24.957273 21755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:24.957480 21755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:24.958619 21755 hybrid_clock.cc:648] HybridClock initialized: now 1790462964958606 us; error 28 us; skew 500 ppm
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:24.960826 21755 init.cc:377] Logged in from keytab as oryx/127.13.148.3@KRBTEST.COM (short username oryx)
I20260926 22:49:24.961206 21755 webserver.cc:460] Webserver started at http://127.13.148.3:38869/ using document root <none> and password file <none>
I20260926 22:49:24.961392 21755 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:24.961437 21755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:24.961552 21755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:24.962572 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "774ec9b9793a4a15a41bace124077bd2"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "92f1b6705f005a6b7ee0beada02edaa9"
server_key_iv: "5fcddcc12aebc9504a0298b775de92c4"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.962894 21755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "774ec9b9793a4a15a41bace124077bd2"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "92f1b6705f005a6b7ee0beada02edaa9"
server_key_iv: "5fcddcc12aebc9504a0298b775de92c4"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.964089 21755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:49:24.964921 21770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.965101 21755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:24.965179 21755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "774ec9b9793a4a15a41bace124077bd2"
format_stamp: "Formatted at 2026-09-26 22:49:24 on dist-test-slave-98hb"
server_key: "92f1b6705f005a6b7ee0beada02edaa9"
server_key_iv: "5fcddcc12aebc9504a0298b775de92c4"
server_key_version: "encryptionkey@0"
I20260926 22:49:24.965245 21755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:24.974052 21755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:24.975613 21755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:24.975739 21755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:24.976231 21755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:24.976274 21755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.976303 21755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:24.976320 21755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:24.982501 21755 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:42645
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:24.982563 21885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:42645 every 8 connection(s)
I20260926 22:49:24.982841 21755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:24.985167 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.976694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:36311 (local address 127.13.148.62:36411)
0926 22:49:24.976749 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:24.976752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:24.976913 (+ 161us) server_negotiation.cc:407] Connection header received
0926 22:49:24.977398 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:24.977399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:24.977412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:24.977463 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:24.978243 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.978971 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.980467 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.980760 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.980819 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.982970 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:24.982971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:24.982972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:24.982979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:24.984191 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.984519 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.984520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.984520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.984563 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.984736 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.984737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.984737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.984823 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:24.984866 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:24.985054 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 22:49:24.985106 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:49:24.985351 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.976660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36411 (local address 127.13.148.3:36311)
0926 22:49:24.976803 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:24.976842 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:24.977306 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:24.977528 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:24.977532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:24.977892 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:24.978151 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.978158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.979071 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.979072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.980361 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.980366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.980833 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.980835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.980957 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.981447 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:24.981455 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:24.982878 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:24.984262 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.984265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.984266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.984463 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.984606 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.984606 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.984606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.984689 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.984864 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:24.984866 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:24.984968 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 22:49:24.985040 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 22:49:24.985149 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260926 22:49:24.985814 21886 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36411
I20260926 22:49:24.985881 21886 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:24.986042 21886 heartbeater.cc:511] Master 127.13.148.62:36411 requested a full tablet report, sending...
I20260926 22:49:24.986104 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 21755
I20260926 22:49:24.986232 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:24.986378 21403 ts_manager.cc:194] Registered new tserver with Master: 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3:42645)
I20260926 22:49:24.986439 13904 external_mini_cluster.cc:1428] Setting key b8db9c5a752a704154ca94878a04f083
I20260926 22:49:24.986840 21403 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:36311
I20260926 22:49:24.987264 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:49:24.992137 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.986763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36411 (local address 127.13.148.3:32805)
0926 22:49:24.986885 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:24.986894 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:24.986959 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:24.987111 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:24.987112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:24.987290 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:24.987348 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.987350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.988169 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.988170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.988639 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:24.988640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.989033 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.989033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:24.989096 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.989451 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:24.989455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:24.989807 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:24.991122 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.991122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.991123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.991339 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.991557 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:24.991558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:24.991558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:24.991620 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:24.991820 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:24.991821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:24.991865 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 22:49:24.991950 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 22:49:24.992041 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260926 22:49:24.992147 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:24.986731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:32805 (local address 127.13.148.62:36411)
0926 22:49:24.986834 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:24.986836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:24.986938 (+ 102us) server_negotiation.cc:407] Connection header received
0926 22:49:24.987025 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:24.987026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:24.987035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:24.987061 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:24.987444 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.988080 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.988705 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:24.988954 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:24.989021 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:24.989874 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:24.989875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:24.989876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:24.989882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:24.991029 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.991435 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.991436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.991436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.991491 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:24.991676 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:24.991676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:24.991677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:24.991766 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:24.991812 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:24.991981 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 22:49:24.992059 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 22:49:24 dist-test-slave-98hb krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 22:49:25.007390 21471 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:25.002654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.13.148.62:36411)
0926 22:49:25.002709 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.002711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.002720 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:25.002790 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.002791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.002801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.002845 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:25.003377 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.003836 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.004538 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.004717 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.004770 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.007324 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44324: BlockingRecv error: recv got EOF from 127.0.0.1:44324 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:25.013713 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.008106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.13.148.62:36411)
0926 22:49:25.008143 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.008145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.008166 (+ 21us) server_negotiation.cc:407] Connection header received
0926 22:49:25.008216 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.008216 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.008223 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.008275 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:25.008649 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.009094 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.009720 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.009947 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.009996 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.011763 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:25.011764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:25.011764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:25.011771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:25.012786 (+ 1015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:25.013064 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:25.013065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:25.013065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:25.013112 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:25.013282 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:25.013283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:25.013283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:25.013376 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:25.013458 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.013555 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.013626 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:49:25.015007 21403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:25.016453 21403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:25.022264 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.018287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46851 (local address 127.0.0.1:46354)
0926 22:49:25.018661 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.018670 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.018716 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.018996 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.018997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.018999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:25.019029 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.019030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.020009 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.020010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.021266 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.021269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.021977 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.021978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.022052 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.022066 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.022129 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.022175 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":97,"threads_started":1}
I20260926 22:49:25.022346 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.018513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.13.148.1:46121)
0926 22:49:25.018776 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.018780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.019069 (+ 289us) server_negotiation.cc:407] Connection header received
0926 22:49:25.019108 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.019110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.019124 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.019188 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:25.019357 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.020336 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.021295 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.022043 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.022116 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.022190 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.022218 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.022250 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":73,"threads_started":1}
I20260926 22:49:25.022575 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.018367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.13.148.3:42645)
0926 22:49:25.018515 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.018520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.018746 (+ 226us) server_negotiation.cc:407] Connection header received
0926 22:49:25.018790 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.018792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.018808 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.018884 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:25.019029 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.020126 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.021332 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.022092 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.022144 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.022205 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.022448 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.022488 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260926 22:49:25.022594 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.018513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.13.148.2:46851)
0926 22:49:25.018859 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.018863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.018871 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:25.018905 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.018907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.018920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.018995 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:25.019162 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.019938 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.021371 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.021907 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.021969 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.022045 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.022462 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.022501 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":76,"threads_started":1}
I20260926 22:49:25.022323 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.018287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46121 (local address 127.0.0.1:36642)
0926 22:49:25.018968 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.018977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.019020 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.019256 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.019257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.019259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:25.019296 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.019298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.020403 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.020404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.021220 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.021222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.022112 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.022113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.022164 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.022167 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.022202 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.022235 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260926 22:49:25.022600 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.018284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42645 (local address 127.0.0.1:51402)
0926 22:49:25.018552 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.018591 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.018658 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.018895 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.018897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.018900 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:25.018948 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.018949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.020203 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.020204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.021268 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.021269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.022237 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.022239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.022314 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.022318 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.022366 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.022405 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":22,"thread_start_us":48,"threads_started":1}
I20260926 22:49:25.023306 21680 tablet_service.cc:1467] Processing CreateTablet for tablet fd559458b5884466854bf0bc8cd24115 (DEFAULT_TABLE table=test-table [id=1a0476d5bd824e388643a73c77f3c175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.023536 21540 tablet_service.cc:1467] Processing CreateTablet for tablet fd559458b5884466854bf0bc8cd24115 (DEFAULT_TABLE table=test-table [id=1a0476d5bd824e388643a73c77f3c175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.023536 21820 tablet_service.cc:1467] Processing CreateTablet for tablet fd559458b5884466854bf0bc8cd24115 (DEFAULT_TABLE table=test-table [id=1a0476d5bd824e388643a73c77f3c175]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.023665 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd559458b5884466854bf0bc8cd24115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.023850 21820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd559458b5884466854bf0bc8cd24115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.023850 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd559458b5884466854bf0bc8cd24115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.026783 21911 tablet_bootstrap.cc:492] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Bootstrap starting.
I20260926 22:49:25.027376 21911 tablet_bootstrap.cc:654] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.027356 21912 tablet_bootstrap.cc:492] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Bootstrap starting.
I20260926 22:49:25.027583 21913 tablet_bootstrap.cc:492] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Bootstrap starting.
I20260926 22:49:25.027691 21911 log.cc:824] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:25.027949 21912 tablet_bootstrap.cc:654] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.028128 21913 tablet_bootstrap.cc:654] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.028247 21912 log.cc:824] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:25.028373 21911 tablet_bootstrap.cc:492] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: No bootstrap required, opened a new log
I20260926 22:49:25.028440 21911 ts_tablet_manager.cc:1397] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:25.028427 21913 log.cc:824] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:25.028867 21912 tablet_bootstrap.cc:492] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: No bootstrap required, opened a new log
I20260926 22:49:25.028928 21912 ts_tablet_manager.cc:1397] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:25.029219 21913 tablet_bootstrap.cc:492] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: No bootstrap required, opened a new log
I20260926 22:49:25.029335 21913 ts_tablet_manager.cc:1397] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:25.030445 21912 raft_consensus.cc:348] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.030570 21912 raft_consensus.cc:374] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.030490 21911 raft_consensus.cc:348] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.030591 21912 raft_consensus.cc:729] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.030635 21911 raft_consensus.cc:374] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.030730 21911 raft_consensus.cc:729] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64de6c63caf046468732d803a8926990, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.030707 21912 consensus_queue.cc:260] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.030867 21911 consensus_queue.cc:260] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.030944 21912 ts_tablet_manager.cc:1428] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:25.031076 21886 heartbeater.cc:503] Master 127.13.148.62:36411 was elected leader, sending a full tablet report...
I20260926 22:49:25.031141 21911 ts_tablet_manager.cc:1428] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:49:25.031255 21746 heartbeater.cc:503] Master 127.13.148.62:36411 was elected leader, sending a full tablet report...
I20260926 22:49:25.031790 21913 raft_consensus.cc:348] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.031935 21913 raft_consensus.cc:374] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.031982 21913 raft_consensus.cc:729] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b5e765f7914ad6a7674adb8d3d96d9, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.032092 21913 consensus_queue.cc:260] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.032322 21913 ts_tablet_manager.cc:1428] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:49:25.032382 21606 heartbeater.cc:503] Master 127.13.148.62:36411 was elected leader, sending a full tablet report...
W20260926 22:49:25.098047 21747 tablet.cc:2369] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:49:25.207122 21607 tablet.cc:2369] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:49:25.233471 21887 tablet.cc:2369] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:25.325083 21917 raft_consensus.cc:482] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:25.325235 21917 raft_consensus.cc:504] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.325587 21917 leader_election.cc:288] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64de6c63caf046468732d803a8926990 (127.13.148.2:46851), 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121)
I20260926 22:49:25.328346 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.325825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46121 (local address 127.13.148.3:42155)
0926 22:49:25.325883 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.325896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.325956 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.326096 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.326097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.326099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:25.326170 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.326172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.326802 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.326803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.327599 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.327601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.328092 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.328092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.328177 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.328180 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.328232 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.328271 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":42}
I20260926 22:49:25.328346 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.325822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:42155 (local address 127.13.148.1:46121)
0926 22:49:25.325908 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.325910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.325938 (+ 28us) server_negotiation.cc:407] Connection header received
0926 22:49:25.326021 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.326022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.326032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.326085 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:25.326232 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.326729 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.327684 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.328025 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.328081 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.328090 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.328248 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.328287 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 22:49:25.328365 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.325852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:42117 (local address 127.13.148.2:46851)
0926 22:49:25.325902 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.325904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.326069 (+ 165us) server_negotiation.cc:407] Connection header received
0926 22:49:25.326105 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.326106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.326114 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.326166 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:25.326297 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.326743 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.327680 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.328035 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.328095 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.328103 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.328269 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.328307 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:49:25.328572 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.325844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46851 (local address 127.13.148.3:42117)
0926 22:49:25.325993 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.326002 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.326063 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.326172 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.326172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.326174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:25.326235 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.326237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.326810 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.326810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.327594 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.327596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.328106 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.328106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.328181 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.328185 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.328443 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.328486 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260926 22:49:25.328655 21560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd559458b5884466854bf0bc8cd24115" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" is_pre_election: true
I20260926 22:49:25.328804 21560 raft_consensus.cc:2454] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 0.
I20260926 22:49:25.328862 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd559458b5884466854bf0bc8cd24115" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de6c63caf046468732d803a8926990" is_pre_election: true
I20260926 22:49:25.328994 21700 raft_consensus.cc:2454] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 0.
I20260926 22:49:25.329028 21772 leader_election.cc:302] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b5e765f7914ad6a7674adb8d3d96d9, 774ec9b9793a4a15a41bace124077bd2; no voters:
I20260926 22:49:25.329139 21917 raft_consensus.cc:2781] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:25.329226 21917 raft_consensus.cc:482] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:25.329248 21917 raft_consensus.cc:3037] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.330289 21917 raft_consensus.cc:504] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.330417 21917 leader_election.cc:288] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 election: Requested vote from peers 64de6c63caf046468732d803a8926990 (127.13.148.2:46851), 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121)
I20260926 22:49:25.330572 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd559458b5884466854bf0bc8cd24115" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de6c63caf046468732d803a8926990"
I20260926 22:49:25.330569 21560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd559458b5884466854bf0bc8cd24115" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9"
I20260926 22:49:25.330660 21700 raft_consensus.cc:3037] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.330662 21560 raft_consensus.cc:3037] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.331315 21700 raft_consensus.cc:2454] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 1.
I20260926 22:49:25.331332 21560 raft_consensus.cc:2454] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 1.
I20260926 22:49:25.331475 21771 leader_election.cc:302] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64de6c63caf046468732d803a8926990, 774ec9b9793a4a15a41bace124077bd2; no voters:
I20260926 22:49:25.331568 21917 raft_consensus.cc:2781] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:25.331699 21917 raft_consensus.cc:686] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 1 LEADER]: Becoming Leader. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Running, Role: LEADER
I20260926 22:49:25.331808 21917 consensus_queue.cc:237] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } }
I20260926 22:49:25.332581 21403 catalog_manager.cc:5495] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:25.372088 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.369897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.13.148.3:42645)
0926 22:49:25.369944 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.369946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.369954 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:25.370027 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.370028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.370048 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.370110 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:25.370309 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.370777 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.371462 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.371638 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.371682 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.371796 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.371892 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.371944 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.371959 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.372015 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:49:25.373644 21700 raft_consensus.cc:1261] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Refusing update from remote peer 774ec9b9793a4a15a41bace124077bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.373662 21560 raft_consensus.cc:1261] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Refusing update from remote peer 774ec9b9793a4a15a41bace124077bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.374006 21921 consensus_queue.cc:1035] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.374143 21917 consensus_queue.cc:1035] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.376386 21925 mvcc.cc:205] Tried to move back new op lower bound from 7333736306168131584 to 7333736305999536128. Current Snapshot: MvccSnapshot[applied={T|T < 7333736306168131584}]
I20260926 22:49:25.378044 21403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:25.378126 21403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:25.378773 21403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: Sending DeleteTablet for 3 replicas of tablet fd559458b5884466854bf0bc8cd24115
I20260926 22:49:25.379030 21680 tablet_service.cc:1514] Processing DeleteTablet for tablet fd559458b5884466854bf0bc8cd24115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46354
I20260926 22:49:25.379084 21540 tablet_service.cc:1514] Processing DeleteTablet for tablet fd559458b5884466854bf0bc8cd24115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36642
I20260926 22:49:25.379259 21820 tablet_service.cc:1514] Processing DeleteTablet for tablet fd559458b5884466854bf0bc8cd24115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51402
I20260926 22:49:25.379283 21403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:25.379418 21403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:25.379470 21935 tablet_replica.cc:330] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: stopping tablet replica
I20260926 22:49:25.379560 21935 raft_consensus.cc:2229] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:25.379660 21935 raft_consensus.cc:2258] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.379814 21936 tablet_replica.cc:330] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: stopping tablet replica
I20260926 22:49:25.379899 21936 raft_consensus.cc:2229] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:25.380007 21936 raft_consensus.cc:2258] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.380491 21540 tablet_service.cc:1467] Processing CreateTablet for tablet 51719f4f046247e98938e2eeb88240dd (DEFAULT_TABLE table=test-table [id=5f1b025867ad49faa39f2570fc1bed45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.380513 21680 tablet_service.cc:1467] Processing CreateTablet for tablet 51719f4f046247e98938e2eeb88240dd (DEFAULT_TABLE table=test-table [id=5f1b025867ad49faa39f2570fc1bed45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.380569 21936 ts_tablet_manager.cc:1905] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.380601 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51719f4f046247e98938e2eeb88240dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.380602 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51719f4f046247e98938e2eeb88240dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.380620 21820 tablet_service.cc:1467] Processing CreateTablet for tablet 51719f4f046247e98938e2eeb88240dd (DEFAULT_TABLE table=test-table [id=5f1b025867ad49faa39f2570fc1bed45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.380728 21820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51719f4f046247e98938e2eeb88240dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.381810 21913 tablet_bootstrap.cc:492] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: Bootstrap starting.
I20260926 22:49:25.381814 21911 tablet_bootstrap.cc:492] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: Bootstrap starting.
I20260926 22:49:25.382160 21911 tablet_bootstrap.cc:654] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.382160 21913 tablet_bootstrap.cc:654] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.382745 21913 tablet_bootstrap.cc:492] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: No bootstrap required, opened a new log
I20260926 22:49:25.382793 21913 ts_tablet_manager.cc:1397] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:25.382840 21912 tablet_bootstrap.cc:492] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: Bootstrap starting.
I20260926 22:49:25.382928 21913 raft_consensus.cc:348] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.383001 21913 raft_consensus.cc:374] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.383021 21913 raft_consensus.cc:729] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b5e765f7914ad6a7674adb8d3d96d9, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.383067 21913 consensus_queue.cc:260] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.383160 21913 ts_tablet_manager.cc:1428] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:25.383220 21912 tablet_bootstrap.cc:654] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.383376 21935 ts_tablet_manager.cc:1905] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.383832 21911 tablet_bootstrap.cc:492] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: No bootstrap required, opened a new log
I20260926 22:49:25.383885 21911 ts_tablet_manager.cc:1397] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:25.384014 21911 raft_consensus.cc:348] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.384083 21911 raft_consensus.cc:374] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.384101 21911 raft_consensus.cc:729] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64de6c63caf046468732d803a8926990, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.384140 21911 consensus_queue.cc:260] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.384240 21911 ts_tablet_manager.cc:1428] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.384680 21912 tablet_bootstrap.cc:492] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: No bootstrap required, opened a new log
I20260926 22:49:25.384728 21912 ts_tablet_manager.cc:1397] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:25.384876 21912 raft_consensus.cc:348] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.384956 21912 raft_consensus.cc:374] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.384984 21912 raft_consensus.cc:729] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.385030 21935 ts_tablet_manager.cc:1918] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:25.385022 21912 consensus_queue.cc:260] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.385092 21935 log.cc:1196] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/fd559458b5884466854bf0bc8cd24115
I20260926 22:49:25.385116 21912 ts_tablet_manager.cc:1428] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.384914 21936 ts_tablet_manager.cc:1918] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:25.385356 21936 log.cc:1196] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/fd559458b5884466854bf0bc8cd24115
I20260926 22:49:25.385607 21936 ts_tablet_manager.cc:1939] T fd559458b5884466854bf0bc8cd24115 P 774ec9b9793a4a15a41bace124077bd2: Deleting consensus metadata
I20260926 22:49:25.385637 21935 ts_tablet_manager.cc:1939] T fd559458b5884466854bf0bc8cd24115 P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting consensus metadata
I20260926 22:49:25.385974 21390 catalog_manager.cc:4841] TS 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3:42645): tablet fd559458b5884466854bf0bc8cd24115 (table test-table [id=1a0476d5bd824e388643a73c77f3c175]) successfully deleted
I20260926 22:49:25.386022 21934 tablet_replica.cc:330] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: stopping tablet replica
I20260926 22:49:25.386014 21388 catalog_manager.cc:4841] TS 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121): tablet fd559458b5884466854bf0bc8cd24115 (table test-table [id=1a0476d5bd824e388643a73c77f3c175]) successfully deleted
I20260926 22:49:25.386111 21934 raft_consensus.cc:2229] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:25.386198 21934 raft_consensus.cc:2258] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.386552 21934 ts_tablet_manager.cc:1905] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.387646 21934 ts_tablet_manager.cc:1918] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:25.387698 21934 log.cc:1196] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/fd559458b5884466854bf0bc8cd24115
I20260926 22:49:25.388126 21934 ts_tablet_manager.cc:1939] T fd559458b5884466854bf0bc8cd24115 P 64de6c63caf046468732d803a8926990: Deleting consensus metadata
I20260926 22:49:25.388466 21387 catalog_manager.cc:4841] TS 64de6c63caf046468732d803a8926990 (127.13.148.2:46851): tablet fd559458b5884466854bf0bc8cd24115 (table test-table [id=1a0476d5bd824e388643a73c77f3c175]) successfully deleted
I20260926 22:49:25.451474 21931 raft_consensus.cc:482] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:25.451582 21931 raft_consensus.cc:504] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.451769 21931 leader_election.cc:288] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121), 64de6c63caf046468732d803a8926990 (127.13.148.2:46851)
I20260926 22:49:25.451958 21560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51719f4f046247e98938e2eeb88240dd" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" is_pre_election: true
I20260926 22:49:25.451952 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51719f4f046247e98938e2eeb88240dd" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de6c63caf046468732d803a8926990" is_pre_election: true
I20260926 22:49:25.452061 21560 raft_consensus.cc:2454] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 0.
I20260926 22:49:25.452067 21700 raft_consensus.cc:2454] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 0.
I20260926 22:49:25.452215 21772 leader_election.cc:302] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b5e765f7914ad6a7674adb8d3d96d9, 774ec9b9793a4a15a41bace124077bd2; no voters:
I20260926 22:49:25.452310 21931 raft_consensus.cc:2781] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:25.452350 21931 raft_consensus.cc:482] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:25.452370 21931 raft_consensus.cc:3037] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.453019 21931 raft_consensus.cc:504] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.453125 21931 leader_election.cc:288] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 election: Requested vote from peers 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121), 64de6c63caf046468732d803a8926990 (127.13.148.2:46851)
I20260926 22:49:25.453289 21560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51719f4f046247e98938e2eeb88240dd" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9"
I20260926 22:49:25.453292 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51719f4f046247e98938e2eeb88240dd" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de6c63caf046468732d803a8926990"
I20260926 22:49:25.453349 21560 raft_consensus.cc:3037] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.453361 21700 raft_consensus.cc:3037] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.453936 21560 raft_consensus.cc:2454] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 1.
I20260926 22:49:25.454103 21772 leader_election.cc:302] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b5e765f7914ad6a7674adb8d3d96d9, 774ec9b9793a4a15a41bace124077bd2; no voters:
I20260926 22:49:25.454180 21931 raft_consensus.cc:2781] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:25.454180 21700 raft_consensus.cc:2454] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 1.
I20260926 22:49:25.454222 21931 raft_consensus.cc:686] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 1 LEADER]: Becoming Leader. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Running, Role: LEADER
I20260926 22:49:25.454267 21931 consensus_queue.cc:237] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.454836 21403 catalog_manager.cc:5495] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } health_report { overall_health: HEALTHY } } }
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462964, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.62@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:25.484666 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.478425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.13.148.62:36411)
0926 22:49:25.478471 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.478474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.478596 (+ 122us) server_negotiation.cc:407] Connection header received
0926 22:49:25.478649 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.478650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.478660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.478706 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:25.479142 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.479620 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.480293 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.480461 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.480514 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.482512 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:25.482513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:25.482514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:25.482520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:25.483692 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:25.484054 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:25.484055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:25.484055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:25.484105 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:25.484271 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:25.484272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:25.484272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:25.484364 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:25.484407 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.484539 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.484602 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:49:25.484774 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.478380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36411 (local address 127.0.0.1:44340)
0926 22:49:25.478531 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.478540 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.478600 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.478715 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.478716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.479016 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:25.479063 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.479066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.479686 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.479687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.480208 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.480210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.480516 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.480517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.480575 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.481029 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:25.481033 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:25.482430 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:25.483756 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:25.483761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:25.483763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:25.483994 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:25.484156 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:25.484156 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:25.484157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:25.484211 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:25.484407 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:25.484412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:25.484461 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.484520 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.484673 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":52,"thread_start_us":88,"threads_started":1}
I20260926 22:49:25.486581 21402 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:44340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 22:49:25.486766 21402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:25.488039 21820 tablet_service.cc:1467] Processing CreateTablet for tablet e9501d95fccc4884b3a1e616062e9f70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de858e063f3944a5b20784dabedb5301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:25.488062 21540 tablet_service.cc:1467] Processing CreateTablet for tablet e9501d95fccc4884b3a1e616062e9f70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de858e063f3944a5b20784dabedb5301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:25.488176 21820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9501d95fccc4884b3a1e616062e9f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.488202 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9501d95fccc4884b3a1e616062e9f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.489152 21912 tablet_bootstrap.cc:492] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: Bootstrap starting.
I20260926 22:49:25.489400 21913 tablet_bootstrap.cc:492] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: Bootstrap starting.
I20260926 22:49:25.489441 21912 tablet_bootstrap.cc:654] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.489513 21680 tablet_service.cc:1467] Processing CreateTablet for tablet e9501d95fccc4884b3a1e616062e9f70 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de858e063f3944a5b20784dabedb5301]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 22:49:25.489630 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9501d95fccc4884b3a1e616062e9f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.489810 21913 tablet_bootstrap.cc:654] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.489992 21912 tablet_bootstrap.cc:492] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: No bootstrap required, opened a new log
I20260926 22:49:25.490029 21912 ts_tablet_manager.cc:1397] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 22:49:25.490123 21912 raft_consensus.cc:348] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.490180 21912 raft_consensus.cc:374] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.490192 21912 raft_consensus.cc:729] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.490227 21912 consensus_queue.cc:260] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.490326 21912 ts_tablet_manager.cc:1428] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.490408 21913 tablet_bootstrap.cc:492] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: No bootstrap required, opened a new log
I20260926 22:49:25.490444 21913 ts_tablet_manager.cc:1397] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:25.490417 21931 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.490479 21931 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:25.490590 21913 raft_consensus.cc:348] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.490651 21913 raft_consensus.cc:374] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.490667 21913 raft_consensus.cc:729] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b5e765f7914ad6a7674adb8d3d96d9, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.490706 21913 consensus_queue.cc:260] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.490784 21913 ts_tablet_manager.cc:1428] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.490844 21919 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.490937 21919 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:25.490952 21911 tablet_bootstrap.cc:492] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: Bootstrap starting.
I20260926 22:49:25.491427 21911 tablet_bootstrap.cc:654] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.491923 21911 tablet_bootstrap.cc:492] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: No bootstrap required, opened a new log
I20260926 22:49:25.491961 21911 ts_tablet_manager.cc:1397] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:25.492064 21911 raft_consensus.cc:348] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.492111 21911 raft_consensus.cc:374] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.492123 21911 raft_consensus.cc:729] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64de6c63caf046468732d803a8926990, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.492158 21911 consensus_queue.cc:260] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.492249 21911 ts_tablet_manager.cc:1428] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.492300 21918 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.492372 21918 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:25.500440 21931 raft_consensus.cc:482] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:25.500494 21931 raft_consensus.cc:504] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.500607 21931 leader_election.cc:288] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121), 64de6c63caf046468732d803a8926990 (127.13.148.2:46851)
I20260926 22:49:25.500733 21560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9501d95fccc4884b3a1e616062e9f70" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" is_pre_election: true
I20260926 22:49:25.500761 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9501d95fccc4884b3a1e616062e9f70" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de6c63caf046468732d803a8926990" is_pre_election: true
I20260926 22:49:25.500818 21700 raft_consensus.cc:2454] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 0.
I20260926 22:49:25.500819 21560 raft_consensus.cc:2454] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 0.
I20260926 22:49:25.500929 21771 leader_election.cc:302] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64de6c63caf046468732d803a8926990, 774ec9b9793a4a15a41bace124077bd2; no voters:
I20260926 22:49:25.501031 21931 raft_consensus.cc:2781] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:25.501073 21931 raft_consensus.cc:482] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:25.501091 21931 raft_consensus.cc:3037] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.501650 21931 raft_consensus.cc:504] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.501801 21931 leader_election.cc:288] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 election: Requested vote from peers 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121), 64de6c63caf046468732d803a8926990 (127.13.148.2:46851)
I20260926 22:49:25.501912 21560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9501d95fccc4884b3a1e616062e9f70" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9"
I20260926 22:49:25.501927 21700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9501d95fccc4884b3a1e616062e9f70" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de6c63caf046468732d803a8926990"
I20260926 22:49:25.501984 21700 raft_consensus.cc:3037] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.501987 21560 raft_consensus.cc:3037] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.502445 21700 raft_consensus.cc:2454] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 1.
I20260926 22:49:25.502538 21560 raft_consensus.cc:2454] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 1.
I20260926 22:49:25.502589 21771 leader_election.cc:302] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64de6c63caf046468732d803a8926990, 774ec9b9793a4a15a41bace124077bd2; no voters:
I20260926 22:49:25.502652 21931 raft_consensus.cc:2781] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:25.502697 21931 raft_consensus.cc:686] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [term 1 LEADER]: Becoming Leader. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Running, Role: LEADER
I20260926 22:49:25.502748 21931 consensus_queue.cc:237] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } }
I20260926 22:49:25.502970 21917 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: TxnStatusTablet state changed. Reason: New leader 774ec9b9793a4a15a41bace124077bd2. Latest consensus state: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.503029 21917 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:25.503161 21951 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 22:49:25.503260 21951 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 22:49:25.503229 21402 catalog_manager.cc:5495] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:25.509680 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.507490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.13.148.3:42645)
0926 22:49:25.507531 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.507534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.507591 (+ 57us) server_negotiation.cc:407] Connection header received
0926 22:49:25.507663 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.507664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.507675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.507736 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:25.507832 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.508340 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.509041 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.509237 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.509296 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.509447 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.509482 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.509524 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.509596 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.509616 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:49:25.509714 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.507436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42645 (local address 127.0.0.1:51416)
0926 22:49:25.507550 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.507560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.507609 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.507735 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.507736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.507737 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:25.507787 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.507790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.508413 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.508414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.508972 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.508974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.509301 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.509302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.509365 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.509381 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.509546 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.509554 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.509593 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.509654 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 22:49:25.510530 21700 raft_consensus.cc:1261] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Refusing update from remote peer 774ec9b9793a4a15a41bace124077bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.510592 21560 raft_consensus.cc:1261] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Refusing update from remote peer 774ec9b9793a4a15a41bace124077bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.510721 21931 consensus_queue.cc:1035] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.510881 21931 consensus_queue.cc:1035] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.511313 21918 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: TxnStatusTablet state changed. Reason: New leader 774ec9b9793a4a15a41bace124077bd2. Latest consensus state: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.511390 21918 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:25.511547 21919 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: TxnStatusTablet state changed. Reason: New leader 774ec9b9793a4a15a41bace124077bd2. Latest consensus state: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.511626 21919 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:25.511873 21924 mvcc.cc:205] Tried to move back new op lower bound from 7333736306729943040 to 7333736306699649024. Current Snapshot: MvccSnapshot[applied={T|T < 7333736306729943040}]
I20260926 22:49:25.512406 21917 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.512478 21917 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:25.512660 21919 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.512738 21919 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 42b5e765f7914ad6a7674adb8d3d96d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:25.513120 21917 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.513188 21917 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 774ec9b9793a4a15a41bace124077bd2: This TxnStatusTablet replica's current role is: LEADER
I20260926 22:49:25.513643 21926 mvcc.cc:205] Tried to move back new op lower bound from 7333736306729943040 to 7333736306699649024. Current Snapshot: MvccSnapshot[applied={T|T < 7333736306729943040 or (T in {7333736306729943040})}]
I20260926 22:49:25.513772 21918 tablet_replica.cc:439] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } }
I20260926 22:49:25.513839 21918 tablet_replica.cc:442] T e9501d95fccc4884b3a1e616062e9f70 P 64de6c63caf046468732d803a8926990: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 22:49:25.517647 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.515276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:60261 (local address 127.13.148.3:42645)
0926 22:49:25.515327 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.515329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.515411 (+ 82us) server_negotiation.cc:407] Connection header received
0926 22:49:25.515448 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.515449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.515458 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.515518 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:25.515591 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.516301 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.516981 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.517269 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.517324 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.517432 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.517461 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.517506 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.517540 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.517555 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:49:25.517656 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.515203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42645 (local address 127.13.148.3:60261)
0926 22:49:25.515354 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.515365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.515411 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.515520 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.515520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.515522 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:25.515558 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.515559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.516363 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.516363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.516919 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.516920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.517330 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.517331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.517364 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.517376 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.517508 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.517509 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.517537 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.517568 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260926 22:49:25.519595 21700 raft_consensus.cc:1261] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Refusing update from remote peer 774ec9b9793a4a15a41bace124077bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.519595 21560 raft_consensus.cc:1261] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Refusing update from remote peer 774ec9b9793a4a15a41bace124077bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.519778 21931 consensus_queue.cc:1035] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.519901 21921 consensus_queue.cc:1035] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.532334 21401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:25.532413 21401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:25.532944 21401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: Sending DeleteTablet for 3 replicas of tablet 51719f4f046247e98938e2eeb88240dd
I20260926 22:49:25.533154 21680 tablet_service.cc:1514] Processing DeleteTablet for tablet 51719f4f046247e98938e2eeb88240dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46354
I20260926 22:49:25.533243 21934 tablet_replica.cc:330] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: stopping tablet replica
I20260926 22:49:25.533236 21819 tablet_service.cc:1514] Processing DeleteTablet for tablet 51719f4f046247e98938e2eeb88240dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51402
I20260926 22:49:25.533288 21934 raft_consensus.cc:2229] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:25.533308 21936 tablet_replica.cc:330] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: stopping tablet replica
I20260926 22:49:25.533318 21934 raft_consensus.cc:2258] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.533354 21936 raft_consensus.cc:2229] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:25.533402 21936 raft_consensus.cc:2258] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.533627 21934 ts_tablet_manager.cc:1905] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.533679 21936 ts_tablet_manager.cc:1905] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.535198 21936 ts_tablet_manager.cc:1918] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:25.535212 21540 tablet_service.cc:1514] Processing DeleteTablet for tablet 51719f4f046247e98938e2eeb88240dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36642
I20260926 22:49:25.535266 21936 log.cc:1196] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/51719f4f046247e98938e2eeb88240dd
I20260926 22:49:25.535455 21936 ts_tablet_manager.cc:1939] T 51719f4f046247e98938e2eeb88240dd P 774ec9b9793a4a15a41bace124077bd2: Deleting consensus metadata
I20260926 22:49:25.535676 21935 tablet_replica.cc:330] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: stopping tablet replica
I20260926 22:49:25.535725 21935 raft_consensus.cc:2229] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:25.535753 21935 raft_consensus.cc:2258] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.535986 21935 ts_tablet_manager.cc:1905] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.537313 21390 catalog_manager.cc:4841] TS 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3:42645): tablet 51719f4f046247e98938e2eeb88240dd (table test-table [id=5f1b025867ad49faa39f2570fc1bed45]) successfully deleted
I20260926 22:49:25.537400 21935 ts_tablet_manager.cc:1918] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:25.537451 21935 log.cc:1196] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/51719f4f046247e98938e2eeb88240dd
I20260926 22:49:25.537587 21934 ts_tablet_manager.cc:1918] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:25.537621 21935 ts_tablet_manager.cc:1939] T 51719f4f046247e98938e2eeb88240dd P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting consensus metadata
I20260926 22:49:25.537636 21934 log.cc:1196] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/51719f4f046247e98938e2eeb88240dd
I20260926 22:49:25.537839 21934 ts_tablet_manager.cc:1939] T 51719f4f046247e98938e2eeb88240dd P 64de6c63caf046468732d803a8926990: Deleting consensus metadata
I20260926 22:49:25.537935 21388 catalog_manager.cc:4841] TS 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121): tablet 51719f4f046247e98938e2eeb88240dd (table test-table [id=5f1b025867ad49faa39f2570fc1bed45]) successfully deleted
I20260926 22:49:25.538094 21387 catalog_manager.cc:4841] TS 64de6c63caf046468732d803a8926990 (127.13.148.2:46851): tablet 51719f4f046247e98938e2eeb88240dd (table test-table [id=5f1b025867ad49faa39f2570fc1bed45]) successfully deleted
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 22:49:25.556329 21471 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:25.551206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.13.148.62:36411)
0926 22:49:25.551274 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.551276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.551359 (+ 83us) server_negotiation.cc:407] Connection header received
0926 22:49:25.551451 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.551452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.551463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.551508 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:25.552043 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.552583 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.553295 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.553481 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.553534 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.556263 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44348: BlockingRecv error: recv got EOF from 127.0.0.1:44348 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM
I20260926 22:49:25.563820 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.557039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.13.148.62:36411)
0926 22:49:25.557075 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.557077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.557207 (+ 130us) server_negotiation.cc:407] Connection header received
0926 22:49:25.557254 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.557255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.557264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.557328 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:25.557757 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.558477 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.559104 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.559396 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.559461 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.561526 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:25.561527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:25.561528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:25.561534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:25.562911 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:25.563187 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:25.563188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:25.563189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:25.563244 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:25.563392 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:25.563392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:25.563393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:25.563508 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:25.563591 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.563654 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.563733 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:25.565096 21401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:25.565266 21401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:25.566535 21540 tablet_service.cc:1467] Processing CreateTablet for tablet ac33090a2ff54075859a0c4a71b0fa46 (DEFAULT_TABLE table=test-table [id=345442ee9c734f8c96241bace01b83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.566566 21819 tablet_service.cc:1467] Processing CreateTablet for tablet ac33090a2ff54075859a0c4a71b0fa46 (DEFAULT_TABLE table=test-table [id=345442ee9c734f8c96241bace01b83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.566665 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac33090a2ff54075859a0c4a71b0fa46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.566666 21819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac33090a2ff54075859a0c4a71b0fa46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.567641 21680 tablet_service.cc:1467] Processing CreateTablet for tablet ac33090a2ff54075859a0c4a71b0fa46 (DEFAULT_TABLE table=test-table [id=345442ee9c734f8c96241bace01b83b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.567762 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac33090a2ff54075859a0c4a71b0fa46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.567984 21912 tablet_bootstrap.cc:492] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: Bootstrap starting.
I20260926 22:49:25.568274 21912 tablet_bootstrap.cc:654] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.568818 21912 tablet_bootstrap.cc:492] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: No bootstrap required, opened a new log
I20260926 22:49:25.568861 21912 ts_tablet_manager.cc:1397] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:25.568912 21911 tablet_bootstrap.cc:492] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: Bootstrap starting.
I20260926 22:49:25.568954 21912 raft_consensus.cc:348] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.569015 21912 raft_consensus.cc:374] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.569033 21912 raft_consensus.cc:729] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.569073 21912 consensus_queue.cc:260] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.569164 21912 ts_tablet_manager.cc:1428] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.569309 21911 tablet_bootstrap.cc:654] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.569895 21913 tablet_bootstrap.cc:492] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: Bootstrap starting.
I20260926 22:49:25.570304 21913 tablet_bootstrap.cc:654] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.570809 21911 tablet_bootstrap.cc:492] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: No bootstrap required, opened a new log
I20260926 22:49:25.570856 21911 ts_tablet_manager.cc:1397] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:25.570904 21913 tablet_bootstrap.cc:492] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: No bootstrap required, opened a new log
I20260926 22:49:25.570956 21913 ts_tablet_manager.cc:1397] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:25.571005 21911 raft_consensus.cc:348] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.571069 21911 raft_consensus.cc:374] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.571086 21911 raft_consensus.cc:729] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64de6c63caf046468732d803a8926990, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.571089 21913 raft_consensus.cc:348] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.571128 21911 consensus_queue.cc:260] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.571156 21913 raft_consensus.cc:374] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.571182 21913 raft_consensus.cc:729] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b5e765f7914ad6a7674adb8d3d96d9, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.571210 21911 ts_tablet_manager.cc:1428] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:25.571218 21913 consensus_queue.cc:260] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.571287 21913 ts_tablet_manager.cc:1428] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:25.608422 21921 raft_consensus.cc:482] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:25.608534 21921 raft_consensus.cc:504] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.608743 21921 leader_election.cc:288] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121), 64de6c63caf046468732d803a8926990 (127.13.148.2:46851)
I20260926 22:49:25.608932 21559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac33090a2ff54075859a0c4a71b0fa46" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" is_pre_election: true
I20260926 22:49:25.608922 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac33090a2ff54075859a0c4a71b0fa46" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de6c63caf046468732d803a8926990" is_pre_election: true
I20260926 22:49:25.609027 21559 raft_consensus.cc:2454] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 0.
I20260926 22:49:25.609038 21699 raft_consensus.cc:2454] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 0.
I20260926 22:49:25.609184 21772 leader_election.cc:302] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b5e765f7914ad6a7674adb8d3d96d9, 774ec9b9793a4a15a41bace124077bd2; no voters:
I20260926 22:49:25.609282 21921 raft_consensus.cc:2781] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:25.609313 21921 raft_consensus.cc:482] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:25.609340 21921 raft_consensus.cc:3037] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.610258 21921 raft_consensus.cc:504] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.610394 21921 leader_election.cc:288] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 election: Requested vote from peers 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121), 64de6c63caf046468732d803a8926990 (127.13.148.2:46851)
I20260926 22:49:25.610538 21559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac33090a2ff54075859a0c4a71b0fa46" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9"
I20260926 22:49:25.610620 21559 raft_consensus.cc:3037] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.611441 21559 raft_consensus.cc:2454] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 1.
I20260926 22:49:25.611559 21699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac33090a2ff54075859a0c4a71b0fa46" candidate_uuid: "774ec9b9793a4a15a41bace124077bd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64de6c63caf046468732d803a8926990"
I20260926 22:49:25.611639 21699 raft_consensus.cc:3037] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.612479 21699 raft_consensus.cc:2454] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774ec9b9793a4a15a41bace124077bd2 in term 1.
I20260926 22:49:25.612641 21771 leader_election.cc:302] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64de6c63caf046468732d803a8926990, 774ec9b9793a4a15a41bace124077bd2; no voters:
I20260926 22:49:25.612769 21921 raft_consensus.cc:2781] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:25.612821 21921 raft_consensus.cc:686] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 1 LEADER]: Becoming Leader. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Running, Role: LEADER
I20260926 22:49:25.612864 21921 consensus_queue.cc:237] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.613330 21400 catalog_manager.cc:5495] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "774ec9b9793a4a15a41bace124077bd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } health_report { overall_health: HEALTHY } } }
I20260926 22:49:25.628288 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.626227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.13.148.3:42645)
0926 22:49:25.626271 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.626273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.626282 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:25.626339 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.626340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.626350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.626397 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:25.626509 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.626987 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.627708 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.627900 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.627954 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.628078 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.628109 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.628148 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.628196 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.628215 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:25.628986 21559 raft_consensus.cc:1261] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Refusing update from remote peer 774ec9b9793a4a15a41bace124077bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.629155 21699 raft_consensus.cc:1261] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Refusing update from remote peer 774ec9b9793a4a15a41bace124077bd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.629324 21917 consensus_queue.cc:1035] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.629431 21921 consensus_queue.cc:1035] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.631745 21400 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:25.631825 21400 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:25.632316 21400 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: Sending DeleteTablet for 3 replicas of tablet ac33090a2ff54075859a0c4a71b0fa46
I20260926 22:49:25.632527 21540 tablet_service.cc:1514] Processing DeleteTablet for tablet ac33090a2ff54075859a0c4a71b0fa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36642
I20260926 22:49:25.632614 21935 tablet_replica.cc:330] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: stopping tablet replica
I20260926 22:49:25.632614 21680 tablet_service.cc:1514] Processing DeleteTablet for tablet ac33090a2ff54075859a0c4a71b0fa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46354
I20260926 22:49:25.632655 21935 raft_consensus.cc:2229] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:25.632689 21935 raft_consensus.cc:2258] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.632670 21819 tablet_service.cc:1514] Processing DeleteTablet for tablet ac33090a2ff54075859a0c4a71b0fa46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51402
I20260926 22:49:25.632762 21934 tablet_replica.cc:330] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: stopping tablet replica
I20260926 22:49:25.632822 21934 raft_consensus.cc:2229] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:25.632865 21934 raft_consensus.cc:2258] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.632929 21936 tablet_replica.cc:330] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: stopping tablet replica
I20260926 22:49:25.632951 21935 ts_tablet_manager.cc:1905] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.632977 21936 raft_consensus.cc:2229] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:25.633028 21936 raft_consensus.cc:2258] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.633232 21936 ts_tablet_manager.cc:1905] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.633176 21400 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:25.633312 21400 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:25.634407 21935 ts_tablet_manager.cc:1918] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:25.634455 21935 log.cc:1196] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/ac33090a2ff54075859a0c4a71b0fa46
I20260926 22:49:25.634542 21936 ts_tablet_manager.cc:1918] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:25.634598 21936 log.cc:1196] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/ac33090a2ff54075859a0c4a71b0fa46
I20260926 22:49:25.634640 21935 ts_tablet_manager.cc:1939] T ac33090a2ff54075859a0c4a71b0fa46 P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting consensus metadata
I20260926 22:49:25.634810 21936 ts_tablet_manager.cc:1939] T ac33090a2ff54075859a0c4a71b0fa46 P 774ec9b9793a4a15a41bace124077bd2: Deleting consensus metadata
I20260926 22:49:25.634943 21388 catalog_manager.cc:4841] TS 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121): tablet ac33090a2ff54075859a0c4a71b0fa46 (table test-table [id=345442ee9c734f8c96241bace01b83b3]) successfully deleted
I20260926 22:49:25.635064 21390 catalog_manager.cc:4841] TS 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3:42645): tablet ac33090a2ff54075859a0c4a71b0fa46 (table test-table [id=345442ee9c734f8c96241bace01b83b3]) successfully deleted
I20260926 22:49:25.635303 21934 ts_tablet_manager.cc:1905] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.635624 21819 tablet_service.cc:1467] Processing CreateTablet for tablet 5e70e2ecf3394b598498db3a4f20371e (DEFAULT_TABLE table=test-table [id=a919d7626d134a75a802e7c281ebcda4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.635637 21680 tablet_service.cc:1467] Processing CreateTablet for tablet 5e70e2ecf3394b598498db3a4f20371e (DEFAULT_TABLE table=test-table [id=a919d7626d134a75a802e7c281ebcda4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.635725 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e70e2ecf3394b598498db3a4f20371e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.635730 21819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e70e2ecf3394b598498db3a4f20371e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.636598 21934 ts_tablet_manager.cc:1918] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:25.636653 21934 log.cc:1196] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/ac33090a2ff54075859a0c4a71b0fa46
I20260926 22:49:25.636754 21911 tablet_bootstrap.cc:492] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: Bootstrap starting.
I20260926 22:49:25.636852 21934 ts_tablet_manager.cc:1939] T ac33090a2ff54075859a0c4a71b0fa46 P 64de6c63caf046468732d803a8926990: Deleting consensus metadata
I20260926 22:49:25.636955 21912 tablet_bootstrap.cc:492] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: Bootstrap starting.
I20260926 22:49:25.637179 21912 tablet_bootstrap.cc:654] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.637199 21387 catalog_manager.cc:4841] TS 64de6c63caf046468732d803a8926990 (127.13.148.2:46851): tablet ac33090a2ff54075859a0c4a71b0fa46 (table test-table [id=345442ee9c734f8c96241bace01b83b3]) successfully deleted
I20260926 22:49:25.637307 21911 tablet_bootstrap.cc:654] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.637548 21540 tablet_service.cc:1467] Processing CreateTablet for tablet 5e70e2ecf3394b598498db3a4f20371e (DEFAULT_TABLE table=test-table [id=a919d7626d134a75a802e7c281ebcda4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:25.637645 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e70e2ecf3394b598498db3a4f20371e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:25.637925 21911 tablet_bootstrap.cc:492] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: No bootstrap required, opened a new log
I20260926 22:49:25.637964 21911 ts_tablet_manager.cc:1397] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:25.638093 21911 raft_consensus.cc:348] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.638149 21911 raft_consensus.cc:374] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.638161 21911 raft_consensus.cc:729] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64de6c63caf046468732d803a8926990, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.638191 21911 consensus_queue.cc:260] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.638295 21911 ts_tablet_manager.cc:1428] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.638584 21913 tablet_bootstrap.cc:492] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: Bootstrap starting.
I20260926 22:49:25.638770 21912 tablet_bootstrap.cc:492] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: No bootstrap required, opened a new log
I20260926 22:49:25.638818 21912 ts_tablet_manager.cc:1397] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260926 22:49:25.638907 21912 raft_consensus.cc:348] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.638953 21912 raft_consensus.cc:374] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.638965 21912 raft_consensus.cc:729] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774ec9b9793a4a15a41bace124077bd2, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.638989 21912 consensus_queue.cc:260] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.639050 21912 ts_tablet_manager.cc:1428] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.639171 21913 tablet_bootstrap.cc:654] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:25.639758 21913 tablet_bootstrap.cc:492] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: No bootstrap required, opened a new log
I20260926 22:49:25.639808 21913 ts_tablet_manager.cc:1397] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:25.639937 21913 raft_consensus.cc:348] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.639986 21913 raft_consensus.cc:374] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:25.640009 21913 raft_consensus.cc:729] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b5e765f7914ad6a7674adb8d3d96d9, State: Initialized, Role: FOLLOWER
I20260926 22:49:25.640053 21913 consensus_queue.cc:260] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.640136 21913 ts_tablet_manager.cc:1428] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:25.648592 21918 raft_consensus.cc:482] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:25.648648 21918 raft_consensus.cc:504] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.648914 21918 leader_election.cc:288] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121), 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3:42645)
I20260926 22:49:25.651610 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.649081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:43049 (local address 127.13.148.1:46121)
0926 22:49:25.649146 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.649148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.649261 (+ 113us) server_negotiation.cc:407] Connection header received
0926 22:49:25.649313 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.649314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.649322 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.649364 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:25.649498 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.650004 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.650958 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.651287 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.651344 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.651353 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.651498 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.651531 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 22:49:25.651618 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.649113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:50065 (local address 127.13.148.3:42645)
0926 22:49:25.649160 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.649162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.649227 (+ 65us) server_negotiation.cc:407] Connection header received
0926 22:49:25.649292 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.649293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.649305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.649350 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:25.649480 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.649982 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.650958 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.651305 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.651365 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.651372 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.651507 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.651539 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:49:25.651638 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.649037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46121 (local address 127.13.148.2:43049)
0926 22:49:25.649192 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.649201 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.649259 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.649371 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.649372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.649373 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:25.649448 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.649450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.650091 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.650092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.650889 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.650891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.651370 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.651371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.651436 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.651440 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.651486 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.651531 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":20,"thread_start_us":63,"threads_started":1}
I20260926 22:49:25.651643 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.649037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42645 (local address 127.13.148.2:50065)
0926 22:49:25.649167 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.649177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.649236 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.649357 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.649358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.649360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:25.649430 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.649432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.650051 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.650051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.650883 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.650885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.651387 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.651389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.651453 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.651456 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.651499 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.651542 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":41,"thread_start_us":45,"threads_started":1}
I20260926 22:49:25.651868 21559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e70e2ecf3394b598498db3a4f20371e" candidate_uuid: "64de6c63caf046468732d803a8926990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" is_pre_election: true
I20260926 22:49:25.651942 21559 raft_consensus.cc:2454] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64de6c63caf046468732d803a8926990 in term 0.
I20260926 22:49:25.652034 21840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e70e2ecf3394b598498db3a4f20371e" candidate_uuid: "64de6c63caf046468732d803a8926990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774ec9b9793a4a15a41bace124077bd2" is_pre_election: true
I20260926 22:49:25.652107 21632 leader_election.cc:302] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b5e765f7914ad6a7674adb8d3d96d9, 64de6c63caf046468732d803a8926990; no voters:
I20260926 22:49:25.652150 21840 raft_consensus.cc:2454] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64de6c63caf046468732d803a8926990 in term 0.
I20260926 22:49:25.652215 21918 raft_consensus.cc:2781] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:25.652258 21918 raft_consensus.cc:482] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:25.652275 21918 raft_consensus.cc:3037] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.652842 21918 raft_consensus.cc:504] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.652956 21918 leader_election.cc:288] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [CANDIDATE]: Term 1 election: Requested vote from peers 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121), 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3:42645)
I20260926 22:49:25.653070 21559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e70e2ecf3394b598498db3a4f20371e" candidate_uuid: "64de6c63caf046468732d803a8926990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b5e765f7914ad6a7674adb8d3d96d9"
I20260926 22:49:25.653143 21559 raft_consensus.cc:3037] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.653179 21840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e70e2ecf3394b598498db3a4f20371e" candidate_uuid: "64de6c63caf046468732d803a8926990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774ec9b9793a4a15a41bace124077bd2"
I20260926 22:49:25.653235 21840 raft_consensus.cc:3037] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:25.653636 21559 raft_consensus.cc:2454] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64de6c63caf046468732d803a8926990 in term 1.
I20260926 22:49:25.653791 21840 raft_consensus.cc:2454] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64de6c63caf046468732d803a8926990 in term 1.
I20260926 22:49:25.653817 21632 leader_election.cc:302] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b5e765f7914ad6a7674adb8d3d96d9, 64de6c63caf046468732d803a8926990; no voters:
I20260926 22:49:25.653898 21918 raft_consensus.cc:2781] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:25.654027 21918 raft_consensus.cc:686] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 1 LEADER]: Becoming Leader. State: Replica: 64de6c63caf046468732d803a8926990, State: Running, Role: LEADER
I20260926 22:49:25.654086 21918 consensus_queue.cc:237] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } }
I20260926 22:49:25.654673 21401 catalog_manager.cc:5495] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64de6c63caf046468732d803a8926990 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "64de6c63caf046468732d803a8926990" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64de6c63caf046468732d803a8926990" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 } health_report { overall_health: UNKNOWN } } }
I20260926 22:49:25.663550 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.661480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.13.148.2:46851)
0926 22:49:25.661521 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.661522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.661531 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:25.661582 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.661583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.661601 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.661663 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:25.661776 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.662220 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.662883 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.663078 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.663127 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.663259 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.663363 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.663423 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.663467 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.663487 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:25.667692 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.665543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:47185 (local address 127.13.148.3:42645)
0926 22:49:25.665598 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.665601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.665678 (+ 77us) server_negotiation.cc:407] Connection header received
0926 22:49:25.665752 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.665753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.665762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.665805 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:25.665919 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.666356 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.667053 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.667295 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.667347 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.667494 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.667525 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.667565 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.667615 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.667630 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:49:25.667749 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.665489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42645 (local address 127.13.148.2:47185)
0926 22:49:25.665620 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.665630 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.665684 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.665806 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.665807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.665808 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:25.665868 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.665870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.666419 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.666420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.666986 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.666987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.667353 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.667354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.667418 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.667429 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.667566 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.667568 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.667612 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.667650 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260926 22:49:25.669445 21559 raft_consensus.cc:1261] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Refusing update from remote peer 64de6c63caf046468732d803a8926990: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.669577 21840 raft_consensus.cc:1261] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Refusing update from remote peer 64de6c63caf046468732d803a8926990: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:25.669667 21985 consensus_queue.cc:1035] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b5e765f7914ad6a7674adb8d3d96d9" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.669955 21985 consensus_queue.cc:1035] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [LEADER]: Connected to new peer: Peer: permanent_uuid: "774ec9b9793a4a15a41bace124077bd2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:25.675452 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.673720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:44865 (local address 127.13.148.2:46851)
0926 22:49:25.673781 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:25.673783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:25.673790 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:25.673807 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:25.673808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:25.673818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:25.673863 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:25.673961 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.674389 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.674997 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.675156 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.675193 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.675282 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.675302 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.675331 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:25.675374 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 22:49:25.675391 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:49:25.675480 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.673588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46851 (local address 127.13.148.3:44865)
0926 22:49:25.673618 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.673628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:25.673660 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:25.673876 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:25.673876 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:25.673878 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 22:49:25.673913 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.673914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.674437 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.674438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.674942 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:25.674943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:25.675198 (+ 255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:25.675199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:25.675231 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:25.675235 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 22:49:25.675332 (+ 97us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 22:49:25.675333 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 22:49:25.675372 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 22:49:25.675412 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260926 22:49:25.681950 21400 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:25.682035 21400 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:25.682789 21400 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b89c7e1748c247939ec16cddf7ab0de7: Sending DeleteTablet for 3 replicas of tablet 5e70e2ecf3394b598498db3a4f20371e
I20260926 22:49:25.683139 21680 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e70e2ecf3394b598498db3a4f20371e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46354
I20260926 22:49:25.683230 21934 tablet_replica.cc:330] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: stopping tablet replica
I20260926 22:49:25.683286 21934 raft_consensus.cc:2229] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:25.683367 21934 raft_consensus.cc:2258] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.683848 21540 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e70e2ecf3394b598498db3a4f20371e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36642
I20260926 22:49:25.683959 21935 tablet_replica.cc:330] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: stopping tablet replica
I20260926 22:49:25.684027 21935 raft_consensus.cc:2229] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:25.684065 21935 raft_consensus.cc:2258] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.684623 21935 ts_tablet_manager.cc:1905] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.685204 21934 ts_tablet_manager.cc:1905] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.683558 21819 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e70e2ecf3394b598498db3a4f20371e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:25 UTC) from {username='oryx', principal='oryx/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51402
I20260926 22:49:25.686780 21936 tablet_replica.cc:330] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: stopping tablet replica
I20260926 22:49:25.686838 21936 raft_consensus.cc:2229] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:25.686880 21936 raft_consensus.cc:2258] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:25.686998 21935 ts_tablet_manager.cc:1918] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:25.687052 21935 log.cc:1196] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/5e70e2ecf3394b598498db3a4f20371e
I20260926 22:49:25.687144 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 21475
I20260926 22:49:25.687263 21935 ts_tablet_manager.cc:1939] T 5e70e2ecf3394b598498db3a4f20371e P 42b5e765f7914ad6a7674adb8d3d96d9: Deleting consensus metadata
W20260926 22:49:25.692207 21388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:46121 (error 108)
I20260926 22:49:25.692498 21934 ts_tablet_manager.cc:1918] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:25.692560 21934 log.cc:1196] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/5e70e2ecf3394b598498db3a4f20371e
I20260926 22:49:25.692739 21934 ts_tablet_manager.cc:1939] T 5e70e2ecf3394b598498db3a4f20371e P 64de6c63caf046468732d803a8926990: Deleting consensus metadata
I20260926 22:49:25.692780 21995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:25.692594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46121 (local address 127.0.0.1:36654)
0926 22:49:25.692693 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:25.692737 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":42,"threads_started":1}
W20260926 22:49:25.692870 21388 catalog_manager.cc:4584] TS 42b5e765f7914ad6a7674adb8d3d96d9 (127.13.148.1:46121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e70e2ecf3394b598498db3a4f20371e: Network error: Client connection negotiation failed: client connection to 127.13.148.1:46121: connect: Connection refused (error 111)
I20260926 22:49:25.693012 21387 catalog_manager.cc:4841] TS 64de6c63caf046468732d803a8926990 (127.13.148.2:46851): tablet 5e70e2ecf3394b598498db3a4f20371e (table test-table [id=a919d7626d134a75a802e7c281ebcda4]) successfully deleted
I20260926 22:49:25.693104 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 21615
I20260926 22:49:25.693250 21936 ts_tablet_manager.cc:1905] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:25.694597 21936 ts_tablet_manager.cc:1918] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 22:49:25.694640 21936 log.cc:1196] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/5e70e2ecf3394b598498db3a4f20371e
I20260926 22:49:25.694808 21936 ts_tablet_manager.cc:1939] T 5e70e2ecf3394b598498db3a4f20371e P 774ec9b9793a4a15a41bace124077bd2: Deleting consensus metadata
I20260926 22:49:25.695256 21390 catalog_manager.cc:4841] TS 774ec9b9793a4a15a41bace124077bd2 (127.13.148.3:42645): tablet 5e70e2ecf3394b598498db3a4f20371e (table test-table [id=a919d7626d134a75a802e7c281ebcda4]) successfully deleted
I20260926 22:49:25.697584 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 21755
I20260926 22:49:25.702280 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 21371
2026-09-26T22:49:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2363 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21998](info): set up 2 sockets
Sep 26 22:49:25 dist-test-slave-98hb krb5kdc[21998](info): commencing operation
krb5kdc: starting...
W20260926 22:49:26.654248 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:26 dist-test-slave-98hb krb5kdc[21998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:26.762809 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35009
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:26.839321 22011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:26.839514 22011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:26.839543 22011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:26.840648 22011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:26.840687 22011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:26.840704 22011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:26.840723 22011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:26.840739 22011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:26.842140 22011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:26.842373 22011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:26.842559 22011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:26.846966 22016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:26.846980 22017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:26.847139 22011 server_base.cc:1034] running on GCE node
W20260926 22:49:26.846967 22019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:26.847510 22011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:26.847774 22011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:26.848910 22011 hybrid_clock.cc:648] HybridClock initialized: now 1790462966848892 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T22:49:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1148 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:26 dist-test-slave-98hb krb5kdc[22024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:26 dist-test-slave-98hb krb5kdc[22024](info): set up 2 sockets
Sep 26 22:49:26 dist-test-slave-98hb krb5kdc[22024](info): commencing operation
krb5kdc: starting...
W20260926 22:49:27.822278 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:27 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:27.928190 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42851
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:42851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:28.004988 22037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:28.005182 22037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:28.005203 22037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:28.006332 22037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:28.006372 22037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:28.006384 22037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:28.006395 22037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:28.006407 22037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:28.007791 22037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:42851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:28.007972 22037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:28.008137 22037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:49:28.012528 22037 server_base.cc:1034] running on GCE node
W20260926 22:49:28.012455 22043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.012465 22042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.012454 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.012871 22037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:28.013125 22037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:28.014262 22037 hybrid_clock.cc:648] HybridClock initialized: now 1790462968014250 us; error 29 us; skew 500 ppm
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:28.016484 22037 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:28.016906 22037 webserver.cc:460] Webserver started at http://127.13.148.62:39507/ using document root <none> and password file <none>
I20260926 22:49:28.017091 22037 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:28.017136 22037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:28.017262 22037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:28.018286 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "dca43148344b41d2ad04de6a41aa5d17"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "83aa2f5dcf441de373b35ce073f59e97"
server_key_iv: "ba363dbac5b21b0197d929e8ef9a5e4e"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.018597 22037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "dca43148344b41d2ad04de6a41aa5d17"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "83aa2f5dcf441de373b35ce073f59e97"
server_key_iv: "ba363dbac5b21b0197d929e8ef9a5e4e"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.019773 22037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:28.020385 22052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.020545 22037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:28.020623 22037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "dca43148344b41d2ad04de6a41aa5d17"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "83aa2f5dcf441de373b35ce073f59e97"
server_key_iv: "ba363dbac5b21b0197d929e8ef9a5e4e"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.020680 22037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:28.030750 22037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:28.032666 22037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:28.032796 22037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:28.036386 22037 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:42851
I20260926 22:49:28.036422 22104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:42851 every 8 connection(s)
I20260926 22:49:28.036721 22037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:28.037467 22105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:28.039618 22105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Bootstrap starting.
I20260926 22:49:28.040148 22105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:28.040403 22105 log.cc:824] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:28.040995 22105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: No bootstrap required, opened a new log
I20260926 22:49:28.042137 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 22037
I20260926 22:49:28.042232 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:28.042397 13904 external_mini_cluster.cc:1428] Setting key a9800577e56e37c9599976ca59dfb4bd
I20260926 22:49:28.042626 22105 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } }
I20260926 22:49:28.042799 22105 raft_consensus.cc:374] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:28.042845 22105 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca43148344b41d2ad04de6a41aa5d17, State: Initialized, Role: FOLLOWER
I20260926 22:49:28.042960 22105 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } }
I20260926 22:49:28.043041 22105 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:28.043071 22105 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:28.043113 22105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:28.044030 22105 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } }
I20260926 22:49:28.044133 22105 leader_election.cc:302] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca43148344b41d2ad04de6a41aa5d17; no voters:
I20260926 22:49:28.044292 22105 leader_election.cc:288] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:28.044344 22110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:28.044468 22110 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 1 LEADER]: Becoming Leader. State: Replica: dca43148344b41d2ad04de6a41aa5d17, State: Running, Role: LEADER
I20260926 22:49:28.044534 22105 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:28.044552 22110 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } }
I20260926 22:49:28.045568 22112 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca43148344b41d2ad04de6a41aa5d17. Latest consensus state: current_term: 1 leader_uuid: "dca43148344b41d2ad04de6a41aa5d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } } }
I20260926 22:49:28.045650 22112 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:28.045792 22111 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dca43148344b41d2ad04de6a41aa5d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } } }
I20260926 22:49:28.045861 22111 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:28.045866 22119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:28.046377 22119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:28.047888 22119 catalog_manager.cc:1347] Generated new cluster ID: 7f8c76c5b9984b2b809f2e7a8f719ee1
I20260926 22:49:28.047936 22119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:28.052210 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.042869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.13.148.62:42851)
0926 22:49:28.043038 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.043047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.043086 (+ 39us) server_negotiation.cc:407] Connection header received
0926 22:49:28.043237 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.043245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.043302 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.043448 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.043970 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.045297 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.046353 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.046889 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.046987 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.049228 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:28.049232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:28.049233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:28.049244 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:28.050730 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:28.051046 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:28.051049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:28.051050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:28.051140 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:28.051318 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:28.051319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:28.051319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:28.051537 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:28.051673 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:28.051774 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 22:49:28.051904 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260926 22:49:28.073940 22119 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:28.074657 22119 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:28.081333 22119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Generated new TSK 0
I20260926 22:49:28.081516 22119 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:28.108377 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42851
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:28.184870 22131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:28.185068 22131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:28.185098 22131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:28.186254 22131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:28.186292 22131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:28.186326 22131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:28.187690 22131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:28.187907 22131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:28.188109 22131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:28.192713 22137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.192685 22136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.192847 22131 server_base.cc:1034] running on GCE node
W20260926 22:49:28.192689 22139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.193153 22131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:28.193383 22131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:28.194514 22131 hybrid_clock.cc:648] HybridClock initialized: now 1790462968194502 us; error 24 us; skew 500 ppm
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:28.196727 22131 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:28.197104 22131 webserver.cc:460] Webserver started at http://127.13.148.1:43367/ using document root <none> and password file <none>
I20260926 22:49:28.197294 22131 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:28.197340 22131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:28.197458 22131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:28.198472 22131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "e0a4cfdd494849f9875d93f250126ad5"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "f95a8ce17d79b1bfa626d2fc0586ab01"
server_key_iv: "8d20afa7d76cb90c8bdeda4272e686fd"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.198786 22131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "e0a4cfdd494849f9875d93f250126ad5"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "f95a8ce17d79b1bfa626d2fc0586ab01"
server_key_iv: "8d20afa7d76cb90c8bdeda4272e686fd"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.199921 22131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:28.200531 22146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.200676 22131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:28.200752 22131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "e0a4cfdd494849f9875d93f250126ad5"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "f95a8ce17d79b1bfa626d2fc0586ab01"
server_key_iv: "8d20afa7d76cb90c8bdeda4272e686fd"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.200809 22131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:28.216293 22131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:28.217983 22131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:28.218108 22131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:28.218304 22131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:28.218582 22131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:28.218614 22131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.218641 22131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:28.218663 22131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.223927 22131 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:33035
I20260926 22:49:28.223973 22259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:33035 every 8 connection(s)
I20260926 22:49:28.224279 22131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:28.232216 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.224909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:53419 (local address 127.13.148.62:42851)
0926 22:49:28.224984 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.224987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.225137 (+ 150us) server_negotiation.cc:407] Connection header received
0926 22:49:28.225584 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.225584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.225597 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.225644 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.226503 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.226997 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.227902 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.228097 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.228149 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.230056 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:28.230058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:28.230058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:28.230064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:28.231109 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:28.231432 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:28.231433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:28.231433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:28.231475 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:28.231673 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:28.231675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:28.231675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:28.231802 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:28.231864 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:28.232084 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 22:49:28.232154 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:49:28.232326 22262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.224885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:53419)
0926 22:49:28.225029 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.225063 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.225513 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.225696 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.225699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.226252 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.226434 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.226439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.227060 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.227060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.227824 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.227826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.228155 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.228156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.228261 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.228712 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.228720 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.229980 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:28.231168 (+ 1188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:28.231173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:28.231175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:28.231379 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:28.231517 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:28.231518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:28.231518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:28.231600 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:28.231858 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:28.231861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:28.231964 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 22:49:28.232038 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 22:49:28.232151 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260926 22:49:28.232672 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 22131
I20260926 22:49:28.232733 22260 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42851
I20260926 22:49:28.232766 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:28.232822 22260 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:28.232934 13904 external_mini_cluster.cc:1428] Setting key d370a6cb57539b958c0cf8d62fac812b
I20260926 22:49:28.232980 22260 heartbeater.cc:511] Master 127.13.148.62:42851 requested a full tablet report, sending...
I20260926 22:49:28.233446 22068 ts_manager.cc:194] Registered new tserver with Master: e0a4cfdd494849f9875d93f250126ad5 (127.13.148.1:33035)
I20260926 22:49:28.234074 22068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:53419
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:28.254671 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42851
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:28.332711 22265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:28.332892 22265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:28.332913 22265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:28.334054 22265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:28.334093 22265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:28.334117 22265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:28.335503 22265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:28.335711 22265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:28.335913 22265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:28.340436 22271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.340446 22273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.340541 22265 server_base.cc:1034] running on GCE node
W20260926 22:49:28.340462 22270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.340834 22265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:28.341032 22265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:28.342202 22265 hybrid_clock.cc:648] HybridClock initialized: now 1790462968342185 us; error 31 us; skew 500 ppm
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:28.344453 22265 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:28.344838 22265 webserver.cc:460] Webserver started at http://127.13.148.2:40815/ using document root <none> and password file <none>
I20260926 22:49:28.345019 22265 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:28.345057 22265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:28.345141 22265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:28.346195 22265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "9b3818a0a85e4f26be8fce2704ef7c54"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "696cb6b1394e85f4624d056d12d9f3a6"
server_key_iv: "1eefb89d63292f0a67141a58e603d1ef"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.346482 22265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "9b3818a0a85e4f26be8fce2704ef7c54"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "696cb6b1394e85f4624d056d12d9f3a6"
server_key_iv: "1eefb89d63292f0a67141a58e603d1ef"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.347604 22265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:28.348209 22280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.348366 22265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:28.348441 22265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "9b3818a0a85e4f26be8fce2704ef7c54"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "696cb6b1394e85f4624d056d12d9f3a6"
server_key_iv: "1eefb89d63292f0a67141a58e603d1ef"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.348508 22265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:28.357100 22265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:28.359005 22265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:28.359126 22265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:28.359319 22265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:28.359577 22265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:28.359609 22265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.359628 22265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:28.359652 22265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.365106 22265 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:44525
I20260926 22:49:28.365144 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:44525 every 8 connection(s)
I20260926 22:49:28.365446 22265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:28.368722 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 22265
I20260926 22:49:28.368842 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:28.369033 13904 external_mini_cluster.cc:1428] Setting key 43469c9b1364afde48672f4738f3d98c
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:28.374768 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.366140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:58335 (local address 127.13.148.62:42851)
0926 22:49:28.366200 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.366202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.366381 (+ 179us) server_negotiation.cc:407] Connection header received
0926 22:49:28.366887 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.366889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.366900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.366951 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.367869 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.368364 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.369621 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.369866 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.369930 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.372193 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:28.372194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:28.372194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:28.372200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:28.373422 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:28.373896 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:28.373897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:28.373897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:28.373945 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:28.374197 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:28.374198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:28.374198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:28.374301 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:28.374345 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:28.374641 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 22:49:28.374703 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:49:28.375027 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.366108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:58335)
0926 22:49:28.366265 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.366303 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.366794 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.367020 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.367025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.367580 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.367779 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.367784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.368442 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.368443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.369550 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.369553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.369955 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.369956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.370074 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.370537 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.370546 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.372110 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:28.373497 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:28.373504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:28.373507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:28.373831 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:28.374001 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:28.374002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:28.374002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:28.374133 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:28.374354 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:28.374359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:28.374517 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 22:49:28.374634 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 22:49:28.374801 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260926 22:49:28.375500 22394 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42851
I20260926 22:49:28.375586 22394 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:28.375753 22394 heartbeater.cc:511] Master 127.13.148.62:42851 requested a full tablet report, sending...
I20260926 22:49:28.376116 22068 ts_manager.cc:194] Registered new tserver with Master: 9b3818a0a85e4f26be8fce2704ef7c54 (127.13.148.2:44525)
I20260926 22:49:28.376466 22068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:58335
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:28.391559 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42851
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:28.468631 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:28.468816 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:28.468837 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:28.469996 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:28.470034 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:28.470059 22399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:28.471412 22399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:28.471577 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:28.471746 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:28.476254 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.476248 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.476249 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.476263 22399 server_base.cc:1034] running on GCE node
I20260926 22:49:28.476605 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:28.476835 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:28.477959 22399 hybrid_clock.cc:648] HybridClock initialized: now 1790462968477943 us; error 32 us; skew 500 ppm
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:28.480173 22399 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:28.480557 22399 webserver.cc:460] Webserver started at http://127.13.148.3:33087/ using document root <none> and password file <none>
I20260926 22:49:28.480753 22399 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:28.480799 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:28.480921 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:28.481940 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "4ab6c9b4bca14db08c4c4708be14a771"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "4cda2ebd14fbac08ad700c3eb21956bc"
server_key_iv: "38b1277efe2d12cce0302367555406e5"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.482244 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "4ab6c9b4bca14db08c4c4708be14a771"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "4cda2ebd14fbac08ad700c3eb21956bc"
server_key_iv: "38b1277efe2d12cce0302367555406e5"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.483390 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:28.483989 22414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.484122 22399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:28.484197 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "4ab6c9b4bca14db08c4c4708be14a771"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "4cda2ebd14fbac08ad700c3eb21956bc"
server_key_iv: "38b1277efe2d12cce0302367555406e5"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.484256 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:28.495043 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:28.496925 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:28.497048 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:28.497234 22399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:28.497495 22399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:28.497527 22399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.497553 22399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:28.497570 22399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.502687 22399 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:45837
I20260926 22:49:28.502743 22527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:45837 every 8 connection(s)
I20260926 22:49:28.503019 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:28.505415 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 22399
I20260926 22:49:28.505517 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:28.505676 13904 external_mini_cluster.cc:1428] Setting key 66f004973ed18622875a261498337c96
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:28.511286 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.503612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58879 (local address 127.13.148.62:42851)
0926 22:49:28.503671 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.503673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.503880 (+ 207us) server_negotiation.cc:407] Connection header received
0926 22:49:28.504371 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.504373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.504385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.504435 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.505348 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.505990 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.507052 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.507258 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.507311 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.509094 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:28.509095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:28.509096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:28.509102 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:28.510133 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:28.510460 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:28.510461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:28.510462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:28.510516 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:28.510713 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:28.510714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:28.510714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:28.510824 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:28.510877 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:28.511116 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 22:49:28.511191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:49:28.511417 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.503611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:58879)
0926 22:49:28.503775 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.503807 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.504281 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.504499 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.504504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.505064 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.505241 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.505245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.506075 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.506076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.506975 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.506977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.507321 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.507322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.507444 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.507799 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.507806 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.509012 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:28.510197 (+ 1185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:28.510203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:28.510205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:28.510398 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:28.510565 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:28.510566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:28.510566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:28.510661 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:28.510875 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:28.510881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:28.511005 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 22:49:28.511088 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 22:49:28.511225 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260926 22:49:28.511811 22528 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42851
I20260926 22:49:28.511891 22528 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:28.512040 22528 heartbeater.cc:511] Master 127.13.148.62:42851 requested a full tablet report, sending...
I20260926 22:49:28.512369 22068 ts_manager.cc:194] Registered new tserver with Master: 4ab6c9b4bca14db08c4c4708be14a771 (127.13.148.3:45837)
I20260926 22:49:28.512735 22068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:58879
I20260926 22:49:28.517305 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
Entry for principal oryx/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab.
WARNING: no policy specified for oryx/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
Entry for principal oryx/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab.
WARNING: no policy specified for oryx/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
Entry for principal oryx/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab.
I20260926 22:49:28.577775 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 22131
I20260926 22:49:28.581799 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 22265
I20260926 22:49:28.585426 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 22399
I20260926 22:49:28.588920 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 22037
I20260926 22:49:28.593029 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42851
--webserver_interface=127.13.148.62
--webserver_port=39507
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:42851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:28.670737 22537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:28.670930 22537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:28.670959 22537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:28.672061 22537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:28.672101 22537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:28.672118 22537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:28.672134 22537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:28.672149 22537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:28.673560 22537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:42851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=39507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:28.673823 22537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:28.674016 22537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:28.678359 22542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.678391 22545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.678463 22537 server_base.cc:1034] running on GCE node
W20260926 22:49:28.678383 22543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.678771 22537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:28.679052 22537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:28.680184 22537 hybrid_clock.cc:648] HybridClock initialized: now 1790462968680163 us; error 35 us; skew 500 ppm
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:28.682539 22537 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:28.682925 22537 webserver.cc:460] Webserver started at http://127.13.148.62:39507/ using document root <none> and password file <none>
I20260926 22:49:28.683117 22537 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:28.683175 22537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:28.684396 22537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:49:28.684947 22552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.685087 22537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:28.685163 22537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "dca43148344b41d2ad04de6a41aa5d17"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "83aa2f5dcf441de373b35ce073f59e97"
server_key_iv: "ba363dbac5b21b0197d929e8ef9a5e4e"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.685478 22537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:28.713996 22537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:28.715660 22537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:28.715791 22537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:28.719398 22537 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:42851
I20260926 22:49:28.719436 22604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:42851 every 8 connection(s)
I20260926 22:49:28.719772 22537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:28.721145 22605 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:49:28.721767 22605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Bootstrap starting.
I20260926 22:49:28.727131 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 22537
I20260926 22:49:28.727479 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:33035
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:42851
--webserver_port=43367
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.1
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:49:28.729774 22605 log.cc:824] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:28.731315 22605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:49:28.731554 22605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Bootstrap complete.
I20260926 22:49:28.733510 22605 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } }
I20260926 22:49:28.733723 22605 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca43148344b41d2ad04de6a41aa5d17, State: Initialized, Role: FOLLOWER
I20260926 22:49:28.733919 22605 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } }
I20260926 22:49:28.733996 22605 raft_consensus.cc:388] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:28.734043 22605 raft_consensus.cc:482] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:28.734089 22605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:49:28.735025 22605 raft_consensus.cc:504] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } }
I20260926 22:49:28.735116 22605 leader_election.cc:302] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dca43148344b41d2ad04de6a41aa5d17; no voters:
I20260926 22:49:28.735275 22605 leader_election.cc:288] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 22:49:28.735317 22609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:49:28.735425 22609 raft_consensus.cc:686] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [term 2 LEADER]: Becoming Leader. State: Replica: dca43148344b41d2ad04de6a41aa5d17, State: Running, Role: LEADER
I20260926 22:49:28.735519 22605 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:28.735505 22609 consensus_queue.cc:237] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } }
I20260926 22:49:28.735800 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dca43148344b41d2ad04de6a41aa5d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } } }
I20260926 22:49:28.735805 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dca43148344b41d2ad04de6a41aa5d17. Latest consensus state: current_term: 2 leader_uuid: "dca43148344b41d2ad04de6a41aa5d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dca43148344b41d2ad04de6a41aa5d17" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42851 } } }
I20260926 22:49:28.735893 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:28.735898 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:28.736536 22614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:28.736982 22614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:28.737102 22614 catalog_manager.cc:1259] Loaded cluster ID: 7f8c76c5b9984b2b809f2e7a8f719ee1
I20260926 22:49:28.737130 22614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:28.738395 22614 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:28.738554 22614 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dca43148344b41d2ad04de6a41aa5d17: Loaded TSK: 0
I20260926 22:49:28.738755 22614 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:49:28.806696 22607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:28.806903 22607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:28.806924 22607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:28.808059 22607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:28.808100 22607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:28.808125 22607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:28.809772 22607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.1.keytab
--principal=oryx/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:33035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=43367
--tserver_master_addrs=127.13.148.62:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:28.810011 22607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:28.810227 22607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:28.814821 22631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.814841 22634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.814839 22632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.815137 22607 server_base.cc:1034] running on GCE node
I20260926 22:49:28.815301 22607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:28.815513 22607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:28.816676 22607 hybrid_clock.cc:648] HybridClock initialized: now 1790462968816663 us; error 35 us; skew 500 ppm
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:28.819193 22607 init.cc:377] Logged in from keytab as oryx/127.13.148.1@KRBTEST.COM (short username oryx)
I20260926 22:49:28.819680 22607 webserver.cc:460] Webserver started at http://127.13.148.1:43367/ using document root <none> and password file <none>
I20260926 22:49:28.819940 22607 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:28.820011 22607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:28.821705 22607 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:28.822340 22641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.822497 22607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:28.822583 22607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "e0a4cfdd494849f9875d93f250126ad5"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "f95a8ce17d79b1bfa626d2fc0586ab01"
server_key_iv: "8d20afa7d76cb90c8bdeda4272e686fd"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.822999 22607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:28.839931 22607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:28.842093 22607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:28.842240 22607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:28.842474 22607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:28.842808 22607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:28.842855 22607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.842892 22607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:28.842916 22607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.849660 22607 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:33035
I20260926 22:49:28.849694 22754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:33035 every 8 connection(s)
I20260926 22:49:28.850085 22607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:28.852315 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 22607
I20260926 22:49:28.852602 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:44525
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:42851
--webserver_port=40815
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.2
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:28.859368 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.850787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:38501)
0926 22:49:28.850951 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.850994 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.851421 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.851920 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.851924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.852601 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.852863 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.852870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.854324 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.854325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.855086 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.855089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.855838 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.855839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.855981 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.856532 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.856541 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.859158 (+ 2617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
W20260926 22:49:28.859864 22755 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:42851 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.62:42851: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
W20260926 22:49:28.859877 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:28.850831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38501 (local address 127.13.148.62:42851)
0926 22:49:28.851566 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.851573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.851598 (+ 25us) server_negotiation.cc:407] Connection header received
0926 22:49:28.851692 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.851696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.851744 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.851834 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.852988 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.854233 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.855169 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.855743 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.855843 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.859295 (+ 3452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:38501: BlockingRecv error: recv got EOF from 127.13.148.1:38501 (error 108)
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":71,"threads_started":1}
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:28.865248 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.860084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:52521)
0926 22:49:28.860124 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.860132 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.860163 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.860526 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.860527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.860660 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.860709 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.860711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.861570 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.861571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.862370 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.862372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.862898 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.862899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.862985 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.863394 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.863399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.865188 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 22:49:28.865348 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:28.860148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:52521 (local address 127.13.148.62:42851)
0926 22:49:28.860416 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.860418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.860425 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:28.860444 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.860445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.860456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.860504 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.860781 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.861496 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.862443 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.862746 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.862880 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.865287 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:52521: BlockingRecv error: recv got EOF from 127.13.148.1:52521 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":87,"threads_started":1}
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:28.870102 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.865702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:39919)
0926 22:49:28.865760 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.865770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.865801 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.865916 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.865916 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.866048 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.866077 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.866078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.866890 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.866891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.867682 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.867684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.868103 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.868103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.868152 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.868515 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.868520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.870057 (+ 1537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 22:49:28.870211 22755 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:49:28.870261 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:28.865780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:39919 (local address 127.13.148.62:42851)
0926 22:49:28.865815 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.865817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.865824 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:28.865843 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.865844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.865851 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.865907 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.866138 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.866827 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.867875 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.868045 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.868096 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.870198 (+ 2102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:39919: BlockingRecv error: recv got EOF from 127.13.148.1:39919 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 22:49:28.934557 22759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:28.934767 22759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:28.934790 22759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:28.936019 22759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:28.936074 22759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:28.936100 22759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:28.937633 22759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.2.keytab
--principal=oryx/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:44525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=40815
--tserver_master_addrs=127.13.148.62:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:28.937867 22759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:28.938052 22759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:28.942550 22766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.942817 22759 server_base.cc:1034] running on GCE node
W20260926 22:49:28.942595 22768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:28.943686 22765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:28.943915 22759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:28.944114 22759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:28.945245 22759 hybrid_clock.cc:648] HybridClock initialized: now 1790462968945229 us; error 32 us; skew 500 ppm
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462968, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:28.947542 22759 init.cc:377] Logged in from keytab as oryx/127.13.148.2@KRBTEST.COM (short username oryx)
I20260926 22:49:28.947976 22759 webserver.cc:460] Webserver started at http://127.13.148.2:40815/ using document root <none> and password file <none>
I20260926 22:49:28.948158 22759 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:28.948206 22759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:28.949443 22759 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:28.949954 22775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.950080 22759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.950155 22759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "9b3818a0a85e4f26be8fce2704ef7c54"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "696cb6b1394e85f4624d056d12d9f3a6"
server_key_iv: "1eefb89d63292f0a67141a58e603d1ef"
server_key_version: "encryptionkey@0"
I20260926 22:49:28.950479 22759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:28.958086 22759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:28.959641 22759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:28.959767 22759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:28.959952 22759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:28.960227 22759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:28.960258 22759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.960278 22759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:28.960290 22759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:28.965773 22759 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:44525
I20260926 22:49:28.965807 22888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:44525 every 8 connection(s)
I20260926 22:49:28.966138 22759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:28.968003 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 22759
I20260926 22:49:28.968271 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:45837
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:42851
--webserver_port=33087
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.148.3
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:28.973789 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.966823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:58427)
0926 22:49:28.966971 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.967013 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.967480 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.967713 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.967716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.968331 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.968579 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.968585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.969255 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.969256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.970485 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.970488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.970828 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.970829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.970958 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.971446 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.971455 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.973597 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
W20260926 22:49:28.973816 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:28.966802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:58427 (local address 127.13.148.62:42851)
0926 22:49:28.966860 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.966862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.967095 (+ 233us) server_negotiation.cc:407] Connection header received
0926 22:49:28.967559 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.967560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.967572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.967656 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.968679 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.969169 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.970562 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.970757 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.970814 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.973752 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:58427: BlockingRecv error: recv got EOF from 127.13.148.2:58427 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 22:49:28.974133 22889 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:42851 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.62:42851: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:28.978684 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.974336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:39839)
0926 22:49:28.974445 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.974457 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.974512 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.974640 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.974640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.974788 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.974841 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.974843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.975411 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.975412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.976216 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.976220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.976512 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.976513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.976581 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.976952 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.976957 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.978619 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 22:49:28.978780 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:28.974392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39839 (local address 127.13.148.62:42851)
0926 22:49:28.974446 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.974448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.974503 (+ 55us) server_negotiation.cc:407] Connection header received
0926 22:49:28.974572 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.974573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.974580 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.974621 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.974901 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.975346 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.976288 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.976456 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.976497 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.978711 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:39839: BlockingRecv error: recv got EOF from 127.13.148.2:39839 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:28 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:28.983031 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:28.978945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:57465)
0926 22:49:28.978992 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:28.979003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:28.979044 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:28.979164 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:28.979164 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:28.979296 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:28.979327 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.979328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.979874 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.979874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.980670 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:28.980672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.980966 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.980967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:28.981017 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.981354 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:28.981358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:28.982961 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 22:49:28.983089 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:28.979033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:57465 (local address 127.13.148.62:42851)
0926 22:49:28.979079 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:28.979081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:28.979087 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:49:28.979104 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:28.979105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:28.979112 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:28.979151 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:28.979382 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.979819 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.980747 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:28.980915 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:28.980958 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:28.983036 (+ 2078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:57465: BlockingRecv error: recv got EOF from 127.13.148.2:57465 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 22:49:28.983234 22889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:49:29.049278 22892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:29.049492 22892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:29.049516 22892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:29.050734 22892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:29.050781 22892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:29.050810 22892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:29.052290 22892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/krb5kdc/oryx_127.13.148.3.keytab
--principal=oryx/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:45837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=33087
--tserver_master_addrs=127.13.148.62:42851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:29.052531 22892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:29.052779 22892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:29.057551 22900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:29.057556 22898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:29.057556 22897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:29.057572 22892 server_base.cc:1034] running on GCE node
I20260926 22:49:29.058007 22892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:29.058276 22892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:29.059422 22892 hybrid_clock.cc:648] HybridClock initialized: now 1790462969059408 us; error 31 us; skew 500 ppm
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462969, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:29.061828 22892 init.cc:377] Logged in from keytab as oryx/127.13.148.3@KRBTEST.COM (short username oryx)
I20260926 22:49:29.062261 22892 webserver.cc:460] Webserver started at http://127.13.148.3:33087/ using document root <none> and password file <none>
I20260926 22:49:29.062474 22892 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:29.062537 22892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:29.063921 22892 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:29.064450 22907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:29.064610 22892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:29.064702 22892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "4ab6c9b4bca14db08c4c4708be14a771"
format_stamp: "Formatted at 2026-09-26 22:49:28 on dist-test-slave-98hb"
server_key: "4cda2ebd14fbac08ad700c3eb21956bc"
server_key_iv: "38b1277efe2d12cce0302367555406e5"
server_key_version: "encryptionkey@0"
I20260926 22:49:29.065032 22892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:29.077028 22892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:29.078872 22892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:29.078986 22892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:29.079154 22892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:29.079393 22892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:29.079421 22892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:29.079442 22892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:29.079457 22892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:29.084689 22892 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:45837
I20260926 22:49:29.084748 23020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:45837 every 8 connection(s)
I20260926 22:49:29.085069 22892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 22:49:29.091960 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:29.085754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:57893 (local address 127.13.148.62:42851)
0926 22:49:29.085804 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:29.085805 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:29.086004 (+ 199us) server_negotiation.cc:407] Connection header received
0926 22:49:29.086524 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:29.086526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:29.086538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:29.086588 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:29.087554 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.088031 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.089035 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.089219 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.089275 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.091905 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:57893: BlockingRecv error: recv got EOF from 127.13.148.3:57893 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:29.092015 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:29.085713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:57893)
0926 22:49:29.085894 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:29.085929 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:29.086425 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:29.086661 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:29.086665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:29.087226 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:29.087463 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.087469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.088133 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.088134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.088949 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.088950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.089289 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.089290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.089418 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.089898 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:29.089906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:29.091816 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
W20260926 22:49:29.092216 23021 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:42851 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.148.62:42851: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
I20260926 22:49:29.092999 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 22892
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:29.096681 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:29.092416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:60463)
0926 22:49:29.092455 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:29.092465 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:29.092495 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:29.092681 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:29.092682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:29.092797 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:29.092839 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.092841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.093521 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.093522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.094077 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.094079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.094394 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.094395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.094458 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.094825 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:29.094830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:29.096608 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 22:49:29.096746 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:29.092553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:60463 (local address 127.13.148.62:42851)
0926 22:49:29.092598 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:29.092600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:29.092606 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:49:29.092622 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:29.092623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:29.092631 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:29.092672 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:29.092899 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.093460 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.094140 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.094340 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.094392 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.096687 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:60463: BlockingRecv error: recv got EOF from 127.13.148.3:60463 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:49:29.100962 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:29.093339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.13.148.62:42851)
0926 22:49:29.093387 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:29.093389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:29.093535 (+ 146us) server_negotiation.cc:407] Connection header received
0926 22:49:29.093594 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:29.093595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:29.093605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:29.093663 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:29.094220 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.094684 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.095412 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.095635 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.095688 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.097000 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:29.097006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:29.097006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:29.097020 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:29.099061 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:29.099433 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:29.099437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:29.099437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:29.099580 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:29.099796 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:29.099797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:29.099797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:29.100120 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:29.100261 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:29.100611 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 22:49:29.100812 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:29.102859 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:29.097118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:57837)
0926 22:49:29.097597 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:29.097608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:29.097661 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:29.097805 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:29.097805 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:29.097925 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:29.097960 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.097962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.098805 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.098806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.099681 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.099684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.100185 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.100186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.100363 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.100831 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:29.100836 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:29.102796 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462}
W20260926 22:49:29.102947 22758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:29.097331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:57837 (local address 127.13.148.62:42851)
0926 22:49:29.097415 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:29.097417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:29.097666 (+ 249us) server_negotiation.cc:407] Connection header received
0926 22:49:29.097713 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:29.097714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:29.097724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:29.097801 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:29.098052 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.098717 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.099800 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.100068 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.100109 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.102890 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:57837: BlockingRecv error: recv got EOF from 127.13.148.3:57837 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 22:49:29.103060 23021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:29.877498 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:29.870536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:44581)
0926 22:49:29.870648 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:29.870657 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:29.870701 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:29.870849 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:29.870850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:29.870998 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:29.871045 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.871047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.871850 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.871851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.872411 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.872413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.872698 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.872699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.872735 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.875748 (+ 3013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:29.875753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:29.877429 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
W20260926 22:49:29.877591 23026 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:29.870594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:44581 (local address 127.13.148.62:42851)
0926 22:49:29.870697 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:29.870698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:29.870706 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:29.870749 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:29.870749 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:29.870760 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:29.870807 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:29.871153 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.871787 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.872469 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.872653 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.872694 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.877523 (+ 4829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:44581: BlockingRecv error: recv got EOF from 127.13.148.1:44581 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":52,"threads_started":1}
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:29 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:29.987434 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:29.983563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:57773)
0926 22:49:29.983669 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:29.983680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:29.983733 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:29.983834 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:29.983835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:29.983981 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:29.984028 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.984029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.984600 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.984600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.985122 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:29.985123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.985394 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.985394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:29.985429 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.985829 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:29.985833 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:29.987365 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
W20260926 22:49:29.987528 23026 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:29.983612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:57773 (local address 127.13.148.62:42851)
0926 22:49:29.983691 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:29.983693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:29.983734 (+ 41us) server_negotiation.cc:407] Connection header received
0926 22:49:29.983782 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:29.983783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:29.983793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:29.983829 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:29.984078 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.984549 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.985177 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:29.985347 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:29.985387 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:29.987449 (+ 2062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:57773: BlockingRecv error: recv got EOF from 127.13.148.2:57773 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 22:49:30 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:30 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:30.107587 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:30.103421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:40957)
0926 22:49:30.103545 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:30.103558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:30.103620 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:30.103740 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:30.103741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:30.103894 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:30.103940 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:30.103942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.104630 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.104631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:30.105248 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:30.105251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.105570 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.105571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:30.105608 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:30.106029 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:30.106033 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:30.107521 (+ 1488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
W20260926 22:49:30.107667 23026 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:30.103513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:40957 (local address 127.13.148.62:42851)
0926 22:49:30.103584 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:30.103586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:30.103607 (+ 21us) server_negotiation.cc:407] Connection header received
0926 22:49:30.103678 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:30.103679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:30.103690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:30.103726 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:30.103990 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.104564 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.105326 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.105512 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.105568 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:30.107606 (+ 2038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:40957: BlockingRecv error: recv got EOF from 127.13.148.3:40957 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 22:49:30 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:30 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:30.882125 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:30.877919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:53045)
0926 22:49:30.878043 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:30.878053 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:30.878099 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:30.878222 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:30.878223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:30.878367 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:30.878414 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:30.878415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.879117 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.879118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:30.879684 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:30.879686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.879995 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.879996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:30.880031 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:30.880432 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:30.880435 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:30.882066 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":52,"threads_started":1}
W20260926 22:49:30.882215 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:30.877987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:53045 (local address 127.13.148.62:42851)
0926 22:49:30.878112 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:30.878114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:30.878122 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:30.878144 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:30.878144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:30.878153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:30.878209 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:30.878465 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.879040 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.879755 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.879942 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.879989 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:30.882144 (+ 2155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:53045: BlockingRecv error: recv got EOF from 127.13.148.1:53045 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
Sep 26 22:49:30 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:30 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:30.991967 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:30.987822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:38121)
0926 22:49:30.987945 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:30.987956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:30.988012 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:30.988118 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:30.988118 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:30.988270 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:30.988319 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:30.988320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.989006 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.989006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:30.989511 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:30.989512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.989813 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.989813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:30.989852 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:30.990263 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:30.990268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:30.991910 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
W20260926 22:49:30.992058 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:30.987881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:38121 (local address 127.13.148.62:42851)
0926 22:49:30.987931 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:30.987933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:30.987998 (+ 65us) server_negotiation.cc:407] Connection header received
0926 22:49:30.988052 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:30.988052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:30.988062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:30.988106 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:30.988366 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.988941 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.989566 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:30.989766 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:30.989807 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:30.991993 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:38121: BlockingRecv error: recv got EOF from 127.13.148.2:38121 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 22:49:31 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:31 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:31.112427 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:31.108033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:46505)
0926 22:49:31.108154 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:31.108166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:31.108222 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:31.108352 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:31.108353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:31.108521 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:31.108567 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:31.108569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.109188 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.109190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:31.109703 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:31.109705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.110094 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.110095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:31.110150 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:31.110567 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:31.110572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:31.112370 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
W20260926 22:49:31.112514 23030 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:31.108151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:46505 (local address 127.13.148.62:42851)
0926 22:49:31.108215 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:31.108217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:31.108226 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:31.108282 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:31.108283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:31.108296 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:31.108342 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:31.108637 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.109115 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.109853 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.110035 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.110091 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:31.112454 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:46505: BlockingRecv error: recv got EOF from 127.13.148.3:46505 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 22:49:31 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:31 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:31.886369 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:31.882519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:41243)
0926 22:49:31.882637 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:31.882650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:31.882703 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:31.882822 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:31.882822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:31.882967 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:31.883014 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:31.883016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.883595 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.883596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:31.884108 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:31.884110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.884392 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.884393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:31.884428 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:31.884792 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:31.884796 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:31.886310 (+ 1514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
W20260926 22:49:31.886464 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:31.882561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:41243 (local address 127.13.148.62:42851)
0926 22:49:31.882703 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:31.882705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:31.882713 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:31.882757 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:31.882757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:31.882767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:31.882816 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:31.883062 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.883545 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.884166 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.884346 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.884388 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:31.886397 (+ 2009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:41243: BlockingRecv error: recv got EOF from 127.13.148.1:41243 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
Sep 26 22:49:31 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:31 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:31.996402 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:31.992398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:59745)
0926 22:49:31.992527 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:31.992539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:31.992598 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:31.992701 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:31.992702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:31.992852 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:31.992899 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:31.992900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.993481 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.993481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:31.994060 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:31.994063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.994350 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.994350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:31.994389 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:31.994767 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:31.994771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:31.996344 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
W20260926 22:49:31.996498 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:31.992468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:59745 (local address 127.13.148.62:42851)
0926 22:49:31.992539 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:31.992541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:31.992583 (+ 42us) server_negotiation.cc:407] Connection header received
0926 22:49:31.992639 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:31.992639 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:31.992649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:31.992697 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:31.992947 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.993419 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.994129 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:31.994302 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:31.994343 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:31.996442 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:59745: BlockingRecv error: recv got EOF from 127.13.148.2:59745 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 22:49:32 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:32 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:32.117375 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:32.112881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:54201)
0926 22:49:32.113017 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:32.113027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:32.113077 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:32.113197 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:32.113198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:32.113387 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:32.113535 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:32.113538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.114222 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.114223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:32.114740 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:32.114741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.115063 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.115064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:32.115105 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:32.115510 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:32.115515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:32.117298 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
W20260926 22:49:32.117441 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:32.112967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:54201 (local address 127.13.148.62:42851)
0926 22:49:32.113039 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:32.113041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:32.113066 (+ 25us) server_negotiation.cc:407] Connection header received
0926 22:49:32.113134 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:32.113135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:32.113146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:32.113184 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:32.113632 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.114164 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.114813 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.114994 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.115044 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:32.117388 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:54201: BlockingRecv error: recv got EOF from 127.13.148.3:54201 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 22:49:32 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:32 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:32.891137 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:32.886807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:46357)
0926 22:49:32.886927 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:32.886937 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:32.886988 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:32.887118 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:32.887119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:32.887281 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:32.887330 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:32.887331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.887925 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.887926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:32.888470 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:32.888472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.888821 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.888822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:32.888864 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:32.889353 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:32.889358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:32.891061 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
W20260926 22:49:32.891219 23038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:32.886863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:46357 (local address 127.13.148.62:42851)
0926 22:49:32.886980 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:32.886982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:32.886990 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:32.887035 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:32.887036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:32.887047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:32.887109 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:32.887373 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.887866 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.888548 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.888762 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.888807 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:32.891160 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:46357: BlockingRecv error: recv got EOF from 127.13.148.1:46357 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
Sep 26 22:49:32 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:32 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:33.001114 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:32.996879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:44745)
0926 22:49:32.996990 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:32.997001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:32.997063 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:32.997178 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:32.997179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:32.997355 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:32.997401 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:32.997403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.998098 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.998099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:32.998626 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:32.998628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.998958 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.998958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:32.999012 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:32.999440 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:32.999445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:33.001039 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260926 22:49:33.001184 23038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:32.996933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:44745 (local address 127.13.148.62:42851)
0926 22:49:32.996992 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:32.996994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:32.997045 (+ 51us) server_negotiation.cc:407] Connection header received
0926 22:49:32.997121 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:32.997122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:32.997133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:32.997173 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:32.997471 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.998029 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.998711 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:32.998898 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:32.998953 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:33.001132 (+ 2179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:44745: BlockingRecv error: recv got EOF from 127.13.148.2:44745 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 22:49:33 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:33 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:33.121969 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:33.117861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:38747)
0926 22:49:33.118002 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:33.118014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:33.118074 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:33.118180 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:33.118181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:33.118352 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:33.118402 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:33.118403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:33.118983 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:33.118983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:33.119505 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:33.119506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:33.119797 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:33.119798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:33.119833 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:33.120212 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:33.120216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:33.121892 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
W20260926 22:49:33.122030 23038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:33.117946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:38747 (local address 127.13.148.62:42851)
0926 22:49:33.118022 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:33.118024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:33.118059 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:49:33.118126 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:33.118127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:33.118138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:33.118173 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:33.118455 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:33.118929 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:33.119572 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:33.119748 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:33.119793 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:33.121978 (+ 2185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:38747: BlockingRecv error: recv got EOF from 127.13.148.3:38747 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 22:49:33 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:33 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:33.895872 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:33.891635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:35491)
0926 22:49:33.891770 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:33.891780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:33.891831 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:33.891960 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:33.891961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:33.892113 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:33.892162 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:33.892163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:33.892814 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:33.892815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:33.893465 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:33.893468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:33.893792 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:33.893793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:33.893830 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:33.894216 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:33.894220 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:33.895814 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
W20260926 22:49:33.895969 23042 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:33.891694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:35491 (local address 127.13.148.62:42851)
0926 22:49:33.891817 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:33.891819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:33.891828 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:33.891876 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:33.891876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:33.891887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:33.891950 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:33.892211 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:33.892740 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:33.893528 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:33.893721 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:33.893793 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:33.895902 (+ 2109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:35491: BlockingRecv error: recv got EOF from 127.13.148.1:35491 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
Sep 26 22:49:34 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:34 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:34.005825 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:34.001594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:47839)
0926 22:49:34.001717 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:34.001750 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:34.001805 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:34.001927 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:34.001928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:34.002093 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:34.002144 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:34.002145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.002773 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.002774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:34.003311 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:34.003312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.003596 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.003597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:34.003633 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:34.004029 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:34.004032 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:34.005713 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
W20260926 22:49:34.005887 23042 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:34.001715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:47839 (local address 127.13.148.62:42851)
0926 22:49:34.001808 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:34.001810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:34.001842 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:49:34.001864 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:34.001865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:34.001874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:34.001914 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:34.002203 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.002683 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.003374 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.003548 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.003591 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:34.005839 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:47839: BlockingRecv error: recv got EOF from 127.13.148.2:47839 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 22:49:34 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:34 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:34.126627 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:34.122364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:50367)
0926 22:49:34.122493 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:34.122505 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:34.122580 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:34.122700 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:34.122701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:34.122887 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:34.122934 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:34.122936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.123582 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.123583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:34.124101 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:34.124103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.124435 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.124436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:34.124487 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:34.124881 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:34.124885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:34.126545 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
W20260926 22:49:34.126700 23042 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:34.122430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50367 (local address 127.13.148.62:42851)
0926 22:49:34.122489 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:34.122491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:34.122567 (+ 76us) server_negotiation.cc:407] Connection header received
0926 22:49:34.122640 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:34.122641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:34.122654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:34.122696 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:34.123002 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.123504 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.124176 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.124376 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.124431 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:34.126647 (+ 2216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:50367: BlockingRecv error: recv got EOF from 127.13.148.3:50367 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 22:49:34 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:34 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:34.904176 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:34.896348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:48505)
0926 22:49:34.896472 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:34.896483 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:34.896555 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:34.896705 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:34.896706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:34.896873 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:34.896922 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:34.896924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.897693 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.897694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:34.898267 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:34.898269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.898566 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.898567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:34.898603 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:34.902493 (+ 3890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:34.902499 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:34.904117 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
W20260926 22:49:34.904282 23046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:34.896427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48505 (local address 127.13.148.62:42851)
0926 22:49:34.896550 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:34.896552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:34.896563 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:49:34.896607 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:34.896607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:34.896618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:34.896668 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:34.897002 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.897625 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.898329 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:34.898516 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:34.898565 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:34.904216 (+ 5651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:48505: BlockingRecv error: recv got EOF from 127.13.148.1:48505 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
Sep 26 22:49:35 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:35 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:35.010607 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:35.006301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:43921)
0926 22:49:35.006427 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:35.006438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:35.006495 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:35.006614 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:35.006615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:35.006787 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:35.006836 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:35.006837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.007431 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.007431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:35.007953 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:35.007954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.008254 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.008254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:35.008292 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:35.008750 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:35.008754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:35.010509 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
W20260926 22:49:35.010669 23046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:35.006392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:43921 (local address 127.13.148.62:42851)
0926 22:49:35.006480 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:35.006481 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:35.006492 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:49:35.006541 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:35.006542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:35.006552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:35.006605 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:35.006887 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.007379 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.008007 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.008204 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.008251 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:35.010617 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:43921: BlockingRecv error: recv got EOF from 127.13.148.2:43921 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 22:49:35 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:35 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:35.131464 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:35.127058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:46763)
0926 22:49:35.127180 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:35.127192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:35.127267 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:35.127393 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:35.127393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:35.127563 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:35.127629 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:35.127632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.128359 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.128359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:35.128911 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:35.128913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.129255 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.129255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:35.129291 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:35.129693 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:35.129697 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:35.131404 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
W20260926 22:49:35.131553 23046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:35.127140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:46763 (local address 127.13.148.62:42851)
0926 22:49:35.127208 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:35.127211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:35.127253 (+ 42us) server_negotiation.cc:407] Connection header received
0926 22:49:35.127328 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:35.127329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:35.127341 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:35.127378 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:35.127742 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.128286 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.128985 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.129198 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.129250 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:35.131493 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:46763: BlockingRecv error: recv got EOF from 127.13.148.3:46763 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 22:49:35 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:35 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:35.908623 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:35.904570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:38475)
0926 22:49:35.904694 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:35.904705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:35.904770 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:35.904891 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:35.904892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:35.905043 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:35.905092 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:35.905093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.905688 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.905689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:35.906235 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:35.906238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.906524 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.906525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:35.906561 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:35.906938 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:35.906942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:35.908548 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
W20260926 22:49:35.908701 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:35.904629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38475 (local address 127.13.148.62:42851)
0926 22:49:35.904742 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:35.904744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:35.904758 (+ 14us) server_negotiation.cc:407] Connection header received
0926 22:49:35.904823 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:35.904824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:35.904835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:35.904880 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:35.905141 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.905637 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.906293 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:35.906476 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:35.906520 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:35.908641 (+ 2121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:38475: BlockingRecv error: recv got EOF from 127.13.148.1:38475 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
Sep 26 22:49:36 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:36 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:36.015142 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:36.011042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:56273)
0926 22:49:36.011177 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:36.011189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:36.011252 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:36.011375 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:36.011375 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:36.011526 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:36.011572 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:36.011574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.012150 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.012151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:36.012657 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:36.012659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.012966 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.012966 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:36.013002 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:36.013379 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:36.013383 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:36.015083 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":54,"threads_started":1}
W20260926 22:49:36.015307 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:36.011128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:56273 (local address 127.13.148.62:42851)
0926 22:49:36.011196 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:36.011197 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:36.011236 (+ 39us) server_negotiation.cc:407] Connection header received
0926 22:49:36.011305 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:36.011306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:36.011315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:36.011358 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:36.011618 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.012097 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.012731 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.012914 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.012961 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:36.015235 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:56273: BlockingRecv error: recv got EOF from 127.13.148.2:56273 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 22:49:36 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:36 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:36.136361 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:36.131931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:58009)
0926 22:49:36.132052 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:36.132062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:36.132115 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:36.132250 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:36.132250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:36.132400 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:36.132449 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:36.132450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.133061 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.133062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:36.133595 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:36.133596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.133930 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.133931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:36.133967 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:36.134388 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:36.134393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:36.136304 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
W20260926 22:49:36.136435 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:36.131994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:58009 (local address 127.13.148.62:42851)
0926 22:49:36.132097 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:36.132098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:36.132108 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:49:36.132168 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:36.132169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:36.132188 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:36.132240 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:36.132502 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.132997 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.133672 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.133878 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.133931 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:36.136377 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:58009: BlockingRecv error: recv got EOF from 127.13.148.3:58009 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 22:49:36 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:36 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:36.913443 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:36.909101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:44263)
0926 22:49:36.909234 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:36.909245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:36.909292 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:36.909426 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:36.909427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:36.909579 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:36.909626 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:36.909628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.910483 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.910484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:36.911017 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:36.911019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.911305 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.911305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:36.911341 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:36.911777 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:36.911782 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:36.913375 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
W20260926 22:49:36.913525 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:36.909204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:44263 (local address 127.13.148.62:42851)
0926 22:49:36.909319 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:36.909320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:36.909327 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:36.909348 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:36.909348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:36.909357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:36.909414 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:36.909678 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.910417 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.911075 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:36.911258 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:36.911300 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:36.913459 (+ 2159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:44263: BlockingRecv error: recv got EOF from 127.13.148.1:44263 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
Sep 26 22:49:37 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:37 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:37.020175 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:37.015614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:36727)
0926 22:49:37.015750 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:37.015762 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:37.015827 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:37.015939 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:37.015939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:37.016102 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:37.016151 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:37.016152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.017160 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.017161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:37.017678 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:37.017679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.018007 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.018008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:37.018046 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:37.018436 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:37.018440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:37.020115 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
W20260926 22:49:37.020287 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:37.015687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:36727 (local address 127.13.148.62:42851)
0926 22:49:37.015754 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:37.015756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:37.015819 (+ 63us) server_negotiation.cc:407] Connection header received
0926 22:49:37.015872 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:37.015873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:37.015883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:37.015927 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:37.016237 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.017081 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.017754 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.017957 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.018004 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:37.020219 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:36727: BlockingRecv error: recv got EOF from 127.13.148.2:36727 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 22:49:37 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:37 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:37.141350 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:37.136856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:34117)
0926 22:49:37.136988 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:37.137000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:37.137080 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:37.137220 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:37.137221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:37.137395 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:37.137440 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:37.137442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.138137 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.138138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:37.138677 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:37.138679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.139025 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.139026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:37.139083 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:37.139506 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:37.139510 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:37.141263 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
W20260926 22:49:37.141427 23054 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:37.136928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:34117 (local address 127.13.148.62:42851)
0926 22:49:37.136992 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:37.136994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:37.137059 (+ 65us) server_negotiation.cc:407] Connection header received
0926 22:49:37.137133 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:37.137134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:37.137147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:37.137197 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:37.137513 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.138051 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.138776 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.138961 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.139017 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:37.141374 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:34117: BlockingRecv error: recv got EOF from 127.13.148.3:34117 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 22:49:37 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:37 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:37.918102 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:37.913892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:39475)
0926 22:49:37.914012 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:37.914022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:37.914070 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:37.914236 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:37.914237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:37.914401 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:37.914449 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:37.914450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.915119 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.915120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:37.915649 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:37.915650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.915983 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.915984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:37.916048 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:37.916451 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:37.916454 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:37.918043 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260926 22:49:37.918202 23058 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:37.913977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:39475 (local address 127.13.148.62:42851)
0926 22:49:37.914127 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:37.914129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:37.914137 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:37.914157 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:37.914157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:37.914167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:37.914227 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:37.914507 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.915016 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.915718 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:37.915911 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:37.915960 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:37.918135 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:39475: BlockingRecv error: recv got EOF from 127.13.148.1:39475 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
Sep 26 22:49:38 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:38 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:38.024914 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:38.020588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:54611)
0926 22:49:38.020710 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:38.020721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:38.020785 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:38.020906 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:38.020907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:38.021082 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:38.021127 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:38.021129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.021779 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.021780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:38.022366 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:38.022369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.022729 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.022730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:38.022783 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:38.023169 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:38.023173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:38.024828 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
W20260926 22:49:38.024976 23058 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:38.020631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:54611 (local address 127.13.148.62:42851)
0926 22:49:38.020676 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:38.020678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:38.020777 (+ 99us) server_negotiation.cc:407] Connection header received
0926 22:49:38.020847 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:38.020848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:38.020859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:38.020901 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:38.021190 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.021677 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.022442 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.022667 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.022723 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:38.024924 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:54611: BlockingRecv error: recv got EOF from 127.13.148.2:54611 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 22:49:38 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:38 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:38.146211 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:38.141875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:52333)
0926 22:49:38.142013 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:38.142025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:38.142095 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:38.142237 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:38.142238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:38.142412 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:38.142458 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:38.142459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.143113 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.143114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:38.143649 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:38.143652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.143986 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.143987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:38.144038 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:38.144467 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:38.144472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:38.146126 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
W20260926 22:49:38.146361 23058 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:38.141953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:52333 (local address 127.13.148.62:42851)
0926 22:49:38.142015 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:38.142017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:38.142077 (+ 60us) server_negotiation.cc:407] Connection header received
0926 22:49:38.142144 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:38.142145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:38.142158 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:38.142207 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:38.142529 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.143032 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.143734 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.143919 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.143973 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:38.146223 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:52333: BlockingRecv error: recv got EOF from 127.13.148.3:52333 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 22:49:38 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:38 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:38.922516 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:38.918522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:52443)
0926 22:49:38.918647 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:38.918658 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:38.918726 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:38.918851 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:38.918852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:38.919003 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:38.919049 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:38.919051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.919638 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.919639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:38.920159 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:38.920160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.920447 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.920448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:38.920482 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:38.920877 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:38.920881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:38.922450 (+ 1569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
W20260926 22:49:38.922606 23062 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:38.918591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:52443 (local address 127.13.148.62:42851)
0926 22:49:38.918708 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:38.918710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:38.918719 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:38.918771 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:38.918772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:38.918784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:38.918840 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:38.919102 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.919586 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.920216 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:38.920397 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:38.920441 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:38.922540 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:52443: BlockingRecv error: recv got EOF from 127.13.148.1:52443 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":50,"threads_started":1}
Sep 26 22:49:39 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:39 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:39.029804 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:39.025363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:43945)
0926 22:49:39.025515 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:39.025527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:39.025603 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:39.025723 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:39.025724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:39.025902 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:39.025955 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:39.025957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.026600 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.026600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:39.027160 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:39.027162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.027491 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.027492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:39.027553 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:39.027946 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:39.027950 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:39.029713 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
W20260926 22:49:39.029896 23062 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:39.025462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:43945 (local address 127.13.148.62:42851)
0926 22:49:39.025514 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:39.025516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:39.025609 (+ 93us) server_negotiation.cc:407] Connection header received
0926 22:49:39.025646 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:39.025647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:39.025657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:39.025707 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:39.026010 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.026539 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.027230 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.027415 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.027463 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:39.029838 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:43945: BlockingRecv error: recv got EOF from 127.13.148.2:43945 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 22:49:39 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:39 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:39.150954 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:39.146647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:42243)
0926 22:49:39.146790 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:39.146803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:39.146853 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:39.146969 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:39.146970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:39.147125 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:39.147173 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:39.147174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.147776 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.147777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:39.148321 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:39.148324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.148608 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.148608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:39.148644 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:39.149081 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:39.149087 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:39.150888 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
W20260926 22:49:39.151023 23062 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:39.146757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:42243 (local address 127.13.148.62:42851)
0926 22:49:39.146825 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:39.146827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:39.146838 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:49:39.146907 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:39.146908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:39.146919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:39.146956 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:39.147237 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.147710 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.148385 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.148560 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.148601 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:39.150969 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:42243: BlockingRecv error: recv got EOF from 127.13.148.3:42243 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 22:49:39 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:39 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:39.931830 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:39.922933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:40369)
0926 22:49:39.923060 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:39.923071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:39.923128 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:39.923248 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:39.923248 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:39.923409 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:39.923453 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:39.923454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.924052 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.924052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:39.924591 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:39.924592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.924887 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.924888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:39.924924 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:39.930108 (+ 5184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:39.930116 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:39.931755 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260926 22:49:39.931908 23066 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:39.923009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:40369 (local address 127.13.148.62:42851)
0926 22:49:39.923106 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:39.923108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:39.923116 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:39.923171 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:39.923172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:39.923181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:39.923235 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:39.923506 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.924000 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.924648 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:39.924837 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:39.924879 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:39.931846 (+ 6967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:40369: BlockingRecv error: recv got EOF from 127.13.148.1:40369 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
Sep 26 22:49:40 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:40 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:40.034487 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:40.030229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:49385)
0926 22:49:40.030346 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:40.030357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:40.030420 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:40.030532 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:40.030532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:40.030687 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:40.030748 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:40.030749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.031415 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.031416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:40.031953 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:40.031955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.032271 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.032272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:40.032308 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:40.032757 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:40.032761 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:40.034417 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
W20260926 22:49:40.034559 23066 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:40.030290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:49385 (local address 127.13.148.62:42851)
0926 22:49:40.030365 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:40.030367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:40.030417 (+ 50us) server_negotiation.cc:407] Connection header received
0926 22:49:40.030462 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:40.030463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:40.030473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:40.030518 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:40.030834 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.031351 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.032014 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.032216 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.032268 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:40.034498 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:49385: BlockingRecv error: recv got EOF from 127.13.148.2:49385 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 22:49:40 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:40 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:40.155699 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:40.151394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:44459)
0926 22:49:40.151514 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:40.151525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:40.151583 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:40.151691 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:40.151692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:40.151861 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:40.151907 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:40.151909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.152488 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.152489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:40.153033 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:40.153034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.153353 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.153353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:40.153396 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:40.153879 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:40.153884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:40.155631 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
W20260926 22:49:40.155763 23066 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:40.151469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:44459 (local address 127.13.148.62:42851)
0926 22:49:40.151531 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:40.151533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:40.151581 (+ 48us) server_negotiation.cc:407] Connection header received
0926 22:49:40.151620 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:40.151621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:40.151631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:40.151680 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:40.151957 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.152436 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.153091 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.153297 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.153345 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:40.155710 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:44459: BlockingRecv error: recv got EOF from 127.13.148.3:44459 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 22:49:40 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:40 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:40.936581 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:40.932284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:51551)
0926 22:49:40.932432 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:40.932445 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:40.932503 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:40.932625 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:40.932626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:40.932787 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:40.932833 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:40.932835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.933439 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.933440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:40.933989 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:40.933991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.934345 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.934346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:40.934382 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:40.934810 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:40.934814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:40.936523 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
W20260926 22:49:40.936703 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:40.932369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:51551 (local address 127.13.148.62:42851)
0926 22:49:40.932488 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:40.932490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:40.932498 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:40.932545 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:40.932546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:40.932556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:40.932609 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:40.932892 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.933382 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.934078 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:40.934280 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:40.934343 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:40.936625 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:51551: BlockingRecv error: recv got EOF from 127.13.148.1:51551 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
Sep 26 22:49:41 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:41 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:41.039637 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:41.034934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:50153)
0926 22:49:41.035088 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:41.035100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:41.035175 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:41.035322 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:41.035323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:41.035571 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:41.035624 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:41.035626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.036289 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.036290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:41.036956 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:41.036959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.037399 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.037399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:41.037443 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:41.037881 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:41.037885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:41.039577 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
W20260926 22:49:41.039803 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:41.035028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:50153 (local address 127.13.148.62:42851)
0926 22:49:41.035091 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:41.035093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:41.035164 (+ 71us) server_negotiation.cc:407] Connection header received
0926 22:49:41.035249 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:41.035250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:41.035262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:41.035306 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:41.035706 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.036202 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.037033 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.037327 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.037390 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:41.039733 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:50153: BlockingRecv error: recv got EOF from 127.13.148.2:50153 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 22:49:41 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:41 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:41.160604 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:41.156180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:50019)
0926 22:49:41.156355 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:41.156366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:41.156427 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:41.156553 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:41.156554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:41.156727 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:41.156770 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:41.156772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.157413 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.157414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:41.157983 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:41.157985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.158329 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.158330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:41.158389 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:41.158845 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:41.158849 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:41.160513 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":101,"threads_started":1}
W20260926 22:49:41.160678 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:41.156308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50019 (local address 127.13.148.62:42851)
0926 22:49:41.156369 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:41.156371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:41.156430 (+ 59us) server_negotiation.cc:407] Connection header received
0926 22:49:41.156475 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:41.156476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:41.156490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:41.156537 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:41.156850 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.157332 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.158076 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.158260 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.158317 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:41.160625 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:50019: BlockingRecv error: recv got EOF from 127.13.148.3:50019 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 22:49:41 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:41 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:41.941238 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:41.937034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:42169)
0926 22:49:41.937150 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:41.937162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:41.937214 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:41.937348 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:41.937348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:41.937528 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:41.937580 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:41.937582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.938226 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.938226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:41.938757 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:41.938759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.939058 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.939058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:41.939094 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:41.939506 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:41.939510 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:41.941153 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
W20260926 22:49:41.941303 23074 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:41.937093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42169 (local address 127.13.148.62:42851)
0926 22:49:41.937216 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:41.937218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:41.937227 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:41.937256 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:41.937257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:41.937270 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:41.937332 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:41.937639 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.938153 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.938820 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:41.939005 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:41.939051 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:41.941245 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:42169: BlockingRecv error: recv got EOF from 127.13.148.1:42169 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
Sep 26 22:49:42 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:42 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:42.044636 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:42.040123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:38109)
0926 22:49:42.040264 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:42.040274 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:42.040324 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:42.040454 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:42.040455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:42.040614 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:42.040660 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:42.040662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.041295 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.041296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:42.041898 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:42.041901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.042272 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.042273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:42.042326 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:42.042776 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:42.042780 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:42.044543 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
W20260926 22:49:42.044713 23074 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:42.040246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:38109 (local address 127.13.148.62:42851)
0926 22:49:42.040305 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:42.040307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:42.040316 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:42.040384 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:42.040385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:42.040395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:42.040441 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:42.040737 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.041238 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.041979 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.042176 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.042249 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:42.044659 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:38109: BlockingRecv error: recv got EOF from 127.13.148.2:38109 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 22:49:42 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:42 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:42.165830 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:42.161140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:40103)
0926 22:49:42.161293 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:42.161305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:42.161381 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:42.161519 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:42.161520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:42.161693 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:42.161777 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:42.161780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.162433 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.162434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:42.162968 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:42.162970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.163349 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.163350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:42.163411 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:42.163858 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:42.163864 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:42.165762 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
W20260926 22:49:42.165892 23074 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:42.161232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:40103 (local address 127.13.148.62:42851)
0926 22:49:42.161300 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:42.161302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:42.161372 (+ 70us) server_negotiation.cc:407] Connection header received
0926 22:49:42.161440 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:42.161441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:42.161454 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:42.161505 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:42.161850 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.162350 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.163055 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.163259 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.163334 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:42.165841 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:40103: BlockingRecv error: recv got EOF from 127.13.148.3:40103 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 22:49:42 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:42 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:42.946003 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:42.941667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:50453)
0926 22:49:42.941796 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:42.941809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:42.941881 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:42.942002 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:42.942003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:42.942161 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:42.942208 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:42.942210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.942812 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.942813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:42.943328 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:42.943329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.943653 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.943653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:42.943689 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:42.944136 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:42.944141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:42.945943 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
W20260926 22:49:42.946096 23078 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:42.941764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:50453 (local address 127.13.148.62:42851)
0926 22:49:42.941890 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:42.941891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:42.941900 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:42.941923 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:42.941923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:42.941933 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:42.941988 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:42.942267 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.942757 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.943402 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:42.943596 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:42.943648 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:42.946025 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:50453: BlockingRecv error: recv got EOF from 127.13.148.1:50453 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
Sep 26 22:49:43 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:43 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:43.049921 23079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:43.045246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:32915)
0926 22:49:43.045378 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:43.045390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:43.045460 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:43.045616 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:43.045617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:43.045844 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:43.045892 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:43.045895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.046688 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.046689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:43.047226 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:43.047227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.047583 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.047584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:43.047647 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:43.048084 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:43.048089 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:43.049862 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
W20260926 22:49:43.050035 23078 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:43.045353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:32915 (local address 127.13.148.62:42851)
0926 22:49:43.045404 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:43.045407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:43.045442 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:49:43.045522 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:43.045523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:43.045535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:43.045580 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:43.045971 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.046595 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.047314 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.047516 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.047569 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:43.049964 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:32915: BlockingRecv error: recv got EOF from 127.13.148.2:32915 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 22:49:43 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:43 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.3@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:43.170492 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:43.166262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.3:46869)
0926 22:49:43.166382 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:43.166401 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:43.166462 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:43.166596 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:43.166597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:43.166774 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:43.166818 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:43.166820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.167463 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.167464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:43.167988 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:43.167990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.168330 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.168331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:43.168383 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:43.168813 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:43.168817 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:43.170417 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260926 22:49:43.170581 23078 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:43.166293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:46869 (local address 127.13.148.62:42851)
0926 22:49:43.166371 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:43.166373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:43.166457 (+ 84us) server_negotiation.cc:407] Connection header received
0926 22:49:43.166523 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:43.166523 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:43.166537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:43.166581 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:43.166893 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.167379 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.168068 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.168265 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.168322 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:43.170527 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.3:46869: BlockingRecv error: recv got EOF from 127.13.148.3:46869 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 22:49:43 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:43 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.1@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:43.950738 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:43.946533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.1:42357)
0926 22:49:43.946709 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:43.946721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:43.946776 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:43.946894 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:43.946894 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:43.947043 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:43.947098 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:43.947100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.947729 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.947730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:43.948265 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:43.948267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.948604 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.948605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:43.948641 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:43.949046 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:43.949050 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:43.950675 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
W20260926 22:49:43.950832 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:43.946615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42357 (local address 127.13.148.62:42851)
0926 22:49:43.946744 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:43.946746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:43.946760 (+ 14us) server_negotiation.cc:407] Connection header received
0926 22:49:43.946826 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:43.946827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:43.946836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:43.946888 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:43.947153 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.947647 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.948341 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:43.948537 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:43.948599 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:43.950768 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.1:42357: BlockingRecv error: recv got EOF from 127.13.148.1:42357 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
Sep 26 22:49:44 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 22:49:44 dist-test-slave-98hb krb5kdc[22024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.148.2@KRBTEST.COM for oryx/127.13.148.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 22:49:44.054734 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.050435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42851 (local address 127.13.148.2:53303)
0926 22:49:44.050578 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:44.050589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:44.050648 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:44.050759 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:44.050759 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:44.050907 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:44.050953 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:44.050955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.051549 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.051549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:44.052060 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:44.052061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.052368 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.052368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:44.052404 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.052892 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:44.052896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:44.054648 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.148.62:42851: Server oryx/127.13.148.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
W20260926 22:49:44.054792 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:44.050550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:53303 (local address 127.13.148.62:42851)
0926 22:49:44.050595 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:44.050597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:44.050634 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:49:44.050692 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:44.050693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:44.050703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:44.050748 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:44.051004 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.051490 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.052127 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.052314 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.052363 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.054741 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.148.2:53303: BlockingRecv error: recv got EOF from 127.13.148.2:53303 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:44.096099 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 22607
I20260926 22:49:44.100029 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 22759
I20260926 22:49:44.103601 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 22892
I20260926 22:49:44.107149 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 22537
2026-09-26T22:49:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17256 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T22:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:44Z Disabled control of system clock
I20260926 22:49:44.133033 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45997
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:45997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 22:49:44.210610 23087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:44.210813 23087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:44.210832 23087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:44.211951 23087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:44.211987 23087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:44.211997 23087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:44.212007 23087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:44.212018 23087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:44.213294 23087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:45997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:44.213495 23087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:44.213745 23087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:44.218552 23095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.218552 23093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.218562 23092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:44.218582 23087 server_base.cc:1034] running on GCE node
I20260926 22:49:44.219009 23087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:44.219310 23087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:44.220449 23087 hybrid_clock.cc:648] HybridClock initialized: now 1790462984220436 us; error 30 us; skew 500 ppm
I20260926 22:49:44.221591 23087 webserver.cc:460] Webserver started at http://127.13.148.62:42501/ using document root <none> and password file <none>
I20260926 22:49:44.221840 23087 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:44.221889 23087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:44.222011 23087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:44.223037 23087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "bde1c6a1edb443709cf3b3e9edc59c14"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "0890ed0287752b45cbb394911aff0830"
server_key_iv: "2afda4c31caf73531d65f564ebf92eb3"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.223362 23087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "bde1c6a1edb443709cf3b3e9edc59c14"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "0890ed0287752b45cbb394911aff0830"
server_key_iv: "2afda4c31caf73531d65f564ebf92eb3"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.224503 23087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:44.225117 23101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.225272 23087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:44.225351 23087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "bde1c6a1edb443709cf3b3e9edc59c14"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "0890ed0287752b45cbb394911aff0830"
server_key_iv: "2afda4c31caf73531d65f564ebf92eb3"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.225417 23087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:44.240398 23087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:44.240690 23087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:44.240820 23087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:44.244297 23087 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:45997
I20260926 22:49:44.244333 23153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:45997 every 8 connection(s)
I20260926 22:49:44.244633 23087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:44.245289 23154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:44.246909 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 23087
I20260926 22:49:44.247002 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:44.247184 13904 external_mini_cluster.cc:1428] Setting key 22bac728ad5f016fe199bebb30d5221a
I20260926 22:49:44.248165 23154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14: Bootstrap starting.
I20260926 22:49:44.248808 23154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:44.249082 23154 log.cc:824] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:44.249781 23154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14: No bootstrap required, opened a new log
I20260926 22:49:44.251566 23154 raft_consensus.cc:348] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1c6a1edb443709cf3b3e9edc59c14" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45997 } }
I20260926 22:49:44.251725 23154 raft_consensus.cc:374] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:44.251763 23154 raft_consensus.cc:729] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde1c6a1edb443709cf3b3e9edc59c14, State: Initialized, Role: FOLLOWER
I20260926 22:49:44.251888 23154 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1c6a1edb443709cf3b3e9edc59c14" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45997 } }
I20260926 22:49:44.251968 23154 raft_consensus.cc:388] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:44.251999 23154 raft_consensus.cc:482] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:44.252046 23154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:44.252418 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.247722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.13.148.62:45997)
0926 22:49:44.247893 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:44.247901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:44.247925 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:49:44.248366 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:44.248375 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:44.248440 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:44.248562 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:44.248782 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.249826 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.250577 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.250973 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.251082 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.251734 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:44.251739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:44.251739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:44.251926 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:44.251959 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:44.251973 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:44.252027 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:44.252212 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 22:49:44.252271 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260926 22:49:44.252663 23154 raft_consensus.cc:504] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1c6a1edb443709cf3b3e9edc59c14" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45997 } }
I20260926 22:49:44.252771 23154 leader_election.cc:302] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde1c6a1edb443709cf3b3e9edc59c14; no voters:
I20260926 22:49:44.252928 23154 leader_election.cc:288] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:44.252977 23159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:44.253115 23159 raft_consensus.cc:686] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [term 1 LEADER]: Becoming Leader. State: Replica: bde1c6a1edb443709cf3b3e9edc59c14, State: Running, Role: LEADER
I20260926 22:49:44.253186 23154 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:44.253193 23159 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1c6a1edb443709cf3b3e9edc59c14" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45997 } }
I20260926 22:49:44.253543 23159 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde1c6a1edb443709cf3b3e9edc59c14. Latest consensus state: current_term: 1 leader_uuid: "bde1c6a1edb443709cf3b3e9edc59c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1c6a1edb443709cf3b3e9edc59c14" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45997 } } }
I20260926 22:49:44.253822 23159 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:44.253749 23160 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bde1c6a1edb443709cf3b3e9edc59c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1c6a1edb443709cf3b3e9edc59c14" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45997 } } }
I20260926 22:49:44.254196 23168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:44.254230 23160 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:44.254654 23168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:44.256306 23168 catalog_manager.cc:1347] Generated new cluster ID: a42389bf53e544daa6836e0f38f3482d
I20260926 22:49:44.256361 23168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:44.266278 23168 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:44.266773 23168 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:44.278566 23168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bde1c6a1edb443709cf3b3e9edc59c14: Generated new TSK 0
I20260926 22:49:44.278751 23168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:49:44.285537 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45997
--builtin_ntp_servers=127.13.148.20:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 22:49:44.364012 23178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:44.364219 23178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:44.364246 23178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:44.365350 23178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:44.365387 23178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:44.365419 23178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:44.366743 23178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:44.366966 23178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:44.367235 23178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:44.372749 23184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.372879 23186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.372747 23183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:44.373008 23178 server_base.cc:1034] running on GCE node
I20260926 22:49:44.373250 23178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:44.373478 23178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:44.374640 23178 hybrid_clock.cc:648] HybridClock initialized: now 1790462984374624 us; error 30 us; skew 500 ppm
I20260926 22:49:44.375823 23178 webserver.cc:460] Webserver started at http://127.13.148.1:37429/ using document root <none> and password file <none>
I20260926 22:49:44.376044 23178 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:44.376096 23178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:44.376231 23178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:44.377308 23178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "3c876dafac6b475ca47f077994c09ed5"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "e048b3302c51dabd0e107e1c5c0989b6"
server_key_iv: "2c90931800d84656af0d0ac32eceffe2"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.377853 23178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "3c876dafac6b475ca47f077994c09ed5"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "e048b3302c51dabd0e107e1c5c0989b6"
server_key_iv: "2c90931800d84656af0d0ac32eceffe2"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.379122 23178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:44.379756 23192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.379937 23178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:44.380017 23178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "3c876dafac6b475ca47f077994c09ed5"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "e048b3302c51dabd0e107e1c5c0989b6"
server_key_iv: "2c90931800d84656af0d0ac32eceffe2"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.380081 23178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:44.391057 23178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:44.391346 23178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:44.391472 23178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:44.391673 23178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:44.391987 23178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:44.392019 23178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.392047 23178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:44.392066 23178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.397347 23178 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:40817
I20260926 22:49:44.397404 23305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:40817 every 8 connection(s)
I20260926 22:49:44.397688 23178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:44.400275 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 23178
I20260926 22:49:44.400377 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:44.400554 13904 external_mini_cluster.cc:1428] Setting key ca62991a067bf097243a54367623a39c
I20260926 22:49:44.401420 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45997
--builtin_ntp_servers=127.13.148.20:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 22:49:44.403030 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.398411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:40905 (local address 127.13.148.62:45997)
0926 22:49:44.398476 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:44.398478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:44.398691 (+ 213us) server_negotiation.cc:407] Connection header received
0926 22:49:44.399149 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:44.399150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:44.399159 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:44.399209 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:44.399809 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.400431 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.401616 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.401908 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.401974 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.402796 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:44.402797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:44.402798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:44.402801 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:44.402815 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:44.402819 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:44.402861 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:44.402948 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 22:49:44.402975 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:49:44.403261 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.398393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45997 (local address 127.13.148.1:40905)
0926 22:49:44.398568 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:44.398600 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:44.399074 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:44.399282 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:44.399285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:44.399293 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:44.399741 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:44.399746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.400513 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.400513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:44.401505 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:44.401508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.401992 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.401992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:44.402099 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.402642 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:44.402642 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:44.402680 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:44.402708 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:44.402874 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:44.402878 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:44.402881 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:44.402944 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 22:49:44.403045 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260926 22:49:44.403836 23306 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45997
I20260926 22:49:44.403927 23306 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:44.404095 23306 heartbeater.cc:511] Master 127.13.148.62:45997 requested a full tablet report, sending...
I20260926 22:49:44.404578 23118 ts_manager.cc:194] Registered new tserver with Master: 3c876dafac6b475ca47f077994c09ed5 (127.13.148.1:40817)
I20260926 22:49:44.405179 23118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.1:40905
W20260926 22:49:44.479477 23309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:44.479681 23309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:44.479700 23309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:44.480821 23309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:44.480862 23309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:44.480888 23309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:44.482220 23309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:44.482434 23309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:44.482750 23309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:44.487468 23317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:44.487454 23309 server_base.cc:1034] running on GCE node
W20260926 22:49:44.487429 23315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.487423 23314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:44.487896 23309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:44.488144 23309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:44.489295 23309 hybrid_clock.cc:648] HybridClock initialized: now 1790462984489287 us; error 29 us; skew 500 ppm
I20260926 22:49:44.490533 23309 webserver.cc:460] Webserver started at http://127.13.148.2:36337/ using document root <none> and password file <none>
I20260926 22:49:44.490761 23309 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:44.490809 23309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:44.490936 23309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:44.491976 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "593187a8e31f4b78b2b34d9b40bdb32b"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "5b7291545b5bd67030433c93c9fed3e0"
server_key_iv: "53ea1520422f2e04428baca2f3f0d9be"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.492312 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "593187a8e31f4b78b2b34d9b40bdb32b"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "5b7291545b5bd67030433c93c9fed3e0"
server_key_iv: "53ea1520422f2e04428baca2f3f0d9be"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.493506 23309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:44.494222 23323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.494379 23309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.494458 23309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "593187a8e31f4b78b2b34d9b40bdb32b"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "5b7291545b5bd67030433c93c9fed3e0"
server_key_iv: "53ea1520422f2e04428baca2f3f0d9be"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.494525 23309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:44.504635 23309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:44.504896 23309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:44.505023 23309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:44.505223 23309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:44.505515 23309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:44.505548 23309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.505575 23309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:44.505592 23309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.510900 23309 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:35309
I20260926 22:49:44.510937 23436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:35309 every 8 connection(s)
I20260926 22:49:44.511241 23309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:44.515962 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.511914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:48251 (local address 127.13.148.62:45997)
0926 22:49:44.511976 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:44.511978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:44.512246 (+ 268us) server_negotiation.cc:407] Connection header received
0926 22:49:44.512701 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:44.512703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:44.512713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:44.512773 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:44.513425 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.513927 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.514874 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.515047 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.515088 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.515769 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:44.515771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:44.515771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:44.515774 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:44.515788 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:44.515791 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:44.515828 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:44.515884 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 22:49:44.515905 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:49:44.516160 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.511891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45997 (local address 127.13.148.2:48251)
0926 22:49:44.512123 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:44.512157 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:44.512627 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:44.512826 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:44.512829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:44.512837 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:44.513361 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:44.513366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.514020 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.514021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:44.514807 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:44.514809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.515101 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.515102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:44.515225 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.515646 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:44.515646 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:44.515672 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:44.515696 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:44.515830 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:44.515832 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:44.515834 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:49:44.515878 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:49:44.515951 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260926 22:49:44.516182 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 23309
I20260926 22:49:44.516268 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:44.516436 13904 external_mini_cluster.cc:1428] Setting key 7158bb7e7171fc5a1a6916b9e3d4f9ca
I20260926 22:49:44.516702 23437 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45997
I20260926 22:49:44.516796 23437 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:44.516970 23437 heartbeater.cc:511] Master 127.13.148.62:45997 requested a full tablet report, sending...
I20260926 22:49:44.517319 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45997
--builtin_ntp_servers=127.13.148.20:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 22:49:44.517314 23118 ts_manager.cc:194] Registered new tserver with Master: 593187a8e31f4b78b2b34d9b40bdb32b (127.13.148.2:35309)
I20260926 22:49:44.517674 23118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.2:48251
W20260926 22:49:44.595383 23440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:44.595580 23440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:44.595608 23440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:44.596781 23440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:44.596817 23440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:44.596850 23440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:44.598239 23440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:45997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:44.598459 23440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:44.598732 23440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:44.603299 23446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.603300 23445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:44.603422 23440 server_base.cc:1034] running on GCE node
W20260926 22:49:44.603300 23448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:44.603673 23440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:44.603868 23440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:44.605000 23440 hybrid_clock.cc:648] HybridClock initialized: now 1790462984604981 us; error 32 us; skew 500 ppm
I20260926 22:49:44.606231 23440 webserver.cc:460] Webserver started at http://127.13.148.3:44345/ using document root <none> and password file <none>
I20260926 22:49:44.606434 23440 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:44.606478 23440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:44.606594 23440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:44.607601 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "b9d008d808444f9a8654b77059a2e913"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "f00705f948fb76465833178b70baa778"
server_key_iv: "e721574784c08d0618855a9c228a24a3"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.607916 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "b9d008d808444f9a8654b77059a2e913"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "f00705f948fb76465833178b70baa778"
server_key_iv: "e721574784c08d0618855a9c228a24a3"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.609068 23440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:44.609689 23454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.609839 23440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:44.609920 23440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "b9d008d808444f9a8654b77059a2e913"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "f00705f948fb76465833178b70baa778"
server_key_iv: "e721574784c08d0618855a9c228a24a3"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.609977 23440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:44.630659 23440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:44.630942 23440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:44.631065 23440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:44.631274 23440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:44.631584 23440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:44.631619 23440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.631639 23440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:44.631660 23440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.636909 23440 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:43363
I20260926 22:49:44.636955 23567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:43363 every 8 connection(s)
I20260926 22:49:44.637243 23440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:44.641752 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 23440
I20260926 22:49:44.641846 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:44.642053 13904 external_mini_cluster.cc:1428] Setting key da2d2fd362d15c6c72193da15a908d52
I20260926 22:49:44.642095 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.637935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:39373 (local address 127.13.148.62:45997)
0926 22:49:44.637983 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:44.637985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:44.638232 (+ 247us) server_negotiation.cc:407] Connection header received
0926 22:49:44.638689 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:44.638690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:44.638700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:44.638758 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:44.639356 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.639832 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.640806 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.640978 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.641020 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.641849 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:44.641850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:44.641851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:44.641855 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:44.641869 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:44.641874 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:44.641917 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:44.642015 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 22:49:44.642037 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:49:44.642359 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.637935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:45997 (local address 127.13.148.3:39373)
0926 22:49:44.638117 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:44.638151 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:44.638617 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:44.638813 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:44.638816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:44.638823 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:44.639297 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:44.639302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.639931 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.639932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:44.640737 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:44.640740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:44.641054 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:44.641055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:44.641170 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:44.641684 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:44.641685 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:44.641718 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:44.641767 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:44.641925 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:44.641932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:44.641935 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:44.642010 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 22:49:44.642119 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260926 22:49:44.642791 23568 heartbeater.cc:347] Connected to a master server at 127.13.148.62:45997
I20260926 22:49:44.642886 23568 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:44.643057 23568 heartbeater.cc:511] Master 127.13.148.62:45997 requested a full tablet report, sending...
I20260926 22:49:44.643389 23117 ts_manager.cc:194] Registered new tserver with Master: b9d008d808444f9a8654b77059a2e913 (127.13.148.3:43363)
I20260926 22:49:44.643744 23117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.3:39373
I20260926 22:49:44.653831 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 22:49:44.655529 23157 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:44.655119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.13.148.62:45997)
0926 22:49:44.655159 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:44.655161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:44.655273 (+ 112us) server_negotiation.cc:407] Connection header received
0926 22:49:44.655314 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:44.655314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:44.655322 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:44.655366 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:44.655469 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37310: BlockingRecv error: recv got EOF from 127.0.0.1:37310 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:49:44.655609 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 23178
I20260926 22:49:44.659595 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 23309
I20260926 22:49:44.663479 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 23440
I20260926 22:49:44.666942 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 23087
2026-09-26T22:49:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (558 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T22:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:44Z Disabled control of system clock
I20260926 22:49:44.690871 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36437
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 22:49:44.768518 23579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:44.768714 23579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:44.768734 23579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:44.769832 23579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:44.769870 23579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:44.769881 23579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:44.769893 23579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:44.769903 23579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:44.771179 23579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:44.771373 23579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:44.771607 23579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:44.775995 23584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.776023 23585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.776023 23587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:44.776222 23579 server_base.cc:1034] running on GCE node
I20260926 22:49:44.776433 23579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:44.776707 23579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:44.777848 23579 hybrid_clock.cc:648] HybridClock initialized: now 1790462984777830 us; error 33 us; skew 500 ppm
I20260926 22:49:44.779023 23579 webserver.cc:460] Webserver started at http://127.13.148.62:35127/ using document root <none> and password file <none>
I20260926 22:49:44.779249 23579 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:44.779294 23579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:44.779417 23579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:44.780424 23579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "dfa6b98af871459289291d313639bef4"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "c2f75b6d2f40e4495cf25e99f9abab55"
server_key_iv: "6a4635f39ade5a8fb8c42b1782285c4d"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.780738 23579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "dfa6b98af871459289291d313639bef4"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "c2f75b6d2f40e4495cf25e99f9abab55"
server_key_iv: "6a4635f39ade5a8fb8c42b1782285c4d"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.781921 23579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:44.782570 23593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.782725 23579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:44.782795 23579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "dfa6b98af871459289291d313639bef4"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "c2f75b6d2f40e4495cf25e99f9abab55"
server_key_iv: "6a4635f39ade5a8fb8c42b1782285c4d"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.782855 23579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:44.786084 23579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:44.786324 23579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:44.786439 23579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:44.789868 23579 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36437
I20260926 22:49:44.789913 23645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36437 every 8 connection(s)
I20260926 22:49:44.790251 23579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:44.791075 23646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:44.793205 23646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4: Bootstrap starting.
I20260926 22:49:44.793804 23646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:44.794070 23646 log.cc:824] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:44.794222 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 23579
I20260926 22:49:44.794312 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:44.794498 13904 external_mini_cluster.cc:1428] Setting key e8dd7147056ace6376d874b3d381817f
I20260926 22:49:44.794755 23646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4: No bootstrap required, opened a new log
I20260926 22:49:44.796628 23646 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa6b98af871459289291d313639bef4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36437 } }
I20260926 22:49:44.796792 23646 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:44.796828 23646 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa6b98af871459289291d313639bef4, State: Initialized, Role: FOLLOWER
I20260926 22:49:44.796974 23646 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa6b98af871459289291d313639bef4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36437 } }
I20260926 22:49:44.797072 23646 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:44.797101 23646 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:44.797166 23646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:44.797816 23646 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa6b98af871459289291d313639bef4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36437 } }
I20260926 22:49:44.797883 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.794975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.13.148.62:36437)
0926 22:49:44.795171 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:44.795180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:44.795215 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:49:44.796598 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:44.796607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:44.796660 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:44.796814 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:44.797420 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:44.797423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:44.797424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:44.797427 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:44.797471 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:44.797481 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:44.797535 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:44.797616 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 22:49:44.797662 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260926 22:49:44.797906 23646 leader_election.cc:302] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfa6b98af871459289291d313639bef4; no voters:
I20260926 22:49:44.798046 23646 leader_election.cc:288] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:44.798218 23651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:44.798280 23646 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:44.798344 23651 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [term 1 LEADER]: Becoming Leader. State: Replica: dfa6b98af871459289291d313639bef4, State: Running, Role: LEADER
I20260926 22:49:44.798429 23651 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa6b98af871459289291d313639bef4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36437 } }
I20260926 22:49:44.798774 23652 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfa6b98af871459289291d313639bef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa6b98af871459289291d313639bef4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36437 } } }
I20260926 22:49:44.798784 23653 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfa6b98af871459289291d313639bef4. Latest consensus state: current_term: 1 leader_uuid: "dfa6b98af871459289291d313639bef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa6b98af871459289291d313639bef4" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36437 } } }
I20260926 22:49:44.798867 23652 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:44.798877 23653 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4 [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:44.799037 23660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:44.799490 23660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:44.800833 23660 catalog_manager.cc:1347] Generated new cluster ID: 7ef23786f0034c09aa394ae433feaf0d
I20260926 22:49:44.800884 23660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:44.815373 23660 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:44.815538 23660 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:44.819868 23660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfa6b98af871459289291d313639bef4: Generated new TSK 0
I20260926 22:49:44.820046 23660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:49:44.822696 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36437
--builtin_ntp_servers=127.13.148.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 22:49:44.900888 23670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:44.901079 23670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:44.901099 23670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:44.902264 23670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:44.902300 23670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:44.902325 23670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:44.903606 23670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:44.903802 23670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:44.904069 23670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:44.909474 23678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.909494 23675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:44.909474 23676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:44.909713 23670 server_base.cc:1034] running on GCE node
I20260926 22:49:44.909960 23670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:44.910161 23670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:44.911291 23670 hybrid_clock.cc:648] HybridClock initialized: now 1790462984911275 us; error 33 us; skew 500 ppm
I20260926 22:49:44.912484 23670 webserver.cc:460] Webserver started at http://127.13.148.1:40847/ using document root <none> and password file <none>
I20260926 22:49:44.912681 23670 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:44.912745 23670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:44.912868 23670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:44.913923 23670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "3fa77ca4b7df401093e44139ac7f488b"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "30e9dd627812fa77001b8fa6a301cc6f"
server_key_iv: "0bbe4d1c613ff1ed459dc35432b28f78"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.914242 23670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "3fa77ca4b7df401093e44139ac7f488b"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "30e9dd627812fa77001b8fa6a301cc6f"
server_key_iv: "0bbe4d1c613ff1ed459dc35432b28f78"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.915419 23670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:44.916098 23684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.916276 23670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:44.916368 23670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "3fa77ca4b7df401093e44139ac7f488b"
format_stamp: "Formatted at 2026-09-26 22:49:44 on dist-test-slave-98hb"
server_key: "30e9dd627812fa77001b8fa6a301cc6f"
server_key_iv: "0bbe4d1c613ff1ed459dc35432b28f78"
server_key_version: "encryptionkey@0"
I20260926 22:49:44.916432 23670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:44.920871 23670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:44.921126 23670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:44.921247 23670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:44.921442 23670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:44.921808 23670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:44.921856 23670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.921882 23670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:44.921904 23670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:44.927151 23670 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:45657
I20260926 22:49:44.927212 23797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:45657 every 8 connection(s)
I20260926 22:49:44.927464 23670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:44.930039 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.928075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48833 (local address 127.13.148.62:36437)
0926 22:49:44.928133 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:44.928135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:44.928362 (+ 227us) server_negotiation.cc:407] Connection header received
0926 22:49:44.928778 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:44.928779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:44.928785 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:44.928836 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:44.929856 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:44.929857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:44.929857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:44.929861 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:44.929875 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:44.929879 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:44.929917 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:44.929977 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 22:49:44.929988 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:49:44.930212 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:44.928057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36437 (local address 127.13.148.1:48833)
0926 22:49:44.928218 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:44.928255 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:44.928691 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:44.928896 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:44.928899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:44.928907 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:44.929680 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:44.929681 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:44.929706 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:44.929770 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:44.929923 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:44.929926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:44.929928 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:49:44.929980 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:49:44.930026 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260926 22:49:44.930687 23798 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36437
I20260926 22:49:44.930768 23798 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:44.930900 23798 heartbeater.cc:511] Master 127.13.148.62:36437 requested a full tablet report, sending...
I20260926 22:49:44.931331 23610 ts_manager.cc:194] Registered new tserver with Master: 3fa77ca4b7df401093e44139ac7f488b (127.13.148.1:45657)
I20260926 22:49:44.937078 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 23670
I20260926 22:49:44.937157 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:44.937337 13904 external_mini_cluster.cc:1428] Setting key 1ac3f7485238d05d2a31a58c892be645
I20260926 22:49:44.938153 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36437
--builtin_ntp_servers=127.13.148.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 22:49:45.014986 23801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:45.015192 23801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:45.015214 23801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:45.016326 23801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:45.016360 23801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:45.016386 23801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:45.017639 23801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:45.017863 23801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:45.018116 23801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:45.022683 23807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:45.022783 23801 server_base.cc:1034] running on GCE node
W20260926 22:49:45.022678 23809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:45.022858 23806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:45.023113 23801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:45.023348 23801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:45.024498 23801 hybrid_clock.cc:648] HybridClock initialized: now 1790462985024485 us; error 30 us; skew 500 ppm
I20260926 22:49:45.028501 23801 webserver.cc:460] Webserver started at http://127.13.148.2:35603/ using document root <none> and password file <none>
I20260926 22:49:45.028690 23801 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:45.028729 23801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:45.028820 23801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:45.029820 23801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "ff276d4bce094bb2b1ddd62ed470fbb2"
format_stamp: "Formatted at 2026-09-26 22:49:45 on dist-test-slave-98hb"
server_key: "6839fc32b1fe6eb2fc5ac52834b7a2ff"
server_key_iv: "738a8f338716e48c0a3ad3a29203a0d7"
server_key_version: "encryptionkey@0"
I20260926 22:49:45.030119 23801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "ff276d4bce094bb2b1ddd62ed470fbb2"
format_stamp: "Formatted at 2026-09-26 22:49:45 on dist-test-slave-98hb"
server_key: "6839fc32b1fe6eb2fc5ac52834b7a2ff"
server_key_iv: "738a8f338716e48c0a3ad3a29203a0d7"
server_key_version: "encryptionkey@0"
I20260926 22:49:45.031220 23801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:45.031847 23815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:45.031998 23801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:49:45.032076 23801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "ff276d4bce094bb2b1ddd62ed470fbb2"
format_stamp: "Formatted at 2026-09-26 22:49:45 on dist-test-slave-98hb"
server_key: "6839fc32b1fe6eb2fc5ac52834b7a2ff"
server_key_iv: "738a8f338716e48c0a3ad3a29203a0d7"
server_key_version: "encryptionkey@0"
I20260926 22:49:45.032160 23801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:45.035348 23801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:45.035575 23801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:45.035696 23801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:45.035880 23801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:45.036197 23801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:45.036230 23801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:45.036257 23801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:45.036273 23801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:45.041486 23801 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:42563
I20260926 22:49:45.041519 23928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:42563 every 8 connection(s)
I20260926 22:49:45.041858 23801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:45.044652 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:45.042497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:35471 (local address 127.13.148.62:36437)
0926 22:49:45.042551 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:45.042553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:45.042763 (+ 210us) server_negotiation.cc:407] Connection header received
0926 22:49:45.043265 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:45.043266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:45.043273 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:45.043323 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:45.044406 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:45.044407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:45.044408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:45.044412 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:45.044428 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:45.044432 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:45.044482 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:45.044561 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 22:49:45.044576 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:49:45.044817 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:45.042480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36437 (local address 127.13.148.2:35471)
0926 22:49:45.042651 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:45.042683 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:45.043161 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:45.043390 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:45.043394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:45.043404 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:45.044262 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:45.044263 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:45.044290 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:45.044316 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:45.044489 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:45.044494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:45.044497 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:45.044565 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 22:49:45.044627 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260926 22:49:45.045208 23929 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36437
I20260926 22:49:45.045300 23929 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:45.045436 23929 heartbeater.cc:511] Master 127.13.148.62:36437 requested a full tablet report, sending...
I20260926 22:49:45.045770 23610 ts_manager.cc:194] Registered new tserver with Master: ff276d4bce094bb2b1ddd62ed470fbb2 (127.13.148.2:42563)
I20260926 22:49:45.052064 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 23801
I20260926 22:49:45.052169 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:45.052354 13904 external_mini_cluster.cc:1428] Setting key 4213d6189bd44498d670ef021e9d88d5
I20260926 22:49:45.053184 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36437
--builtin_ntp_servers=127.13.148.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 22:49:45.131078 23932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:45.131283 23932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:45.131312 23932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:45.132460 23932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:45.132498 23932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:45.132532 23932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:45.133850 23932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:45.134085 23932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:45.134397 23932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:45.139223 23937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:45.139225 23940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:45.139253 23938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:45.139269 23932 server_base.cc:1034] running on GCE node
I20260926 22:49:45.139627 23932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:45.139861 23932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:45.141000 23932 hybrid_clock.cc:648] HybridClock initialized: now 1790462985140987 us; error 28 us; skew 500 ppm
I20260926 22:49:45.142205 23932 webserver.cc:460] Webserver started at http://127.13.148.3:40533/ using document root <none> and password file <none>
I20260926 22:49:45.142436 23932 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:45.142482 23932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:45.142601 23932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:45.143620 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "fb237c9d47274161bfa31e7585948461"
format_stamp: "Formatted at 2026-09-26 22:49:45 on dist-test-slave-98hb"
server_key: "ebb932677fdf0fcbe1a9d491ae8e8e65"
server_key_iv: "90f9eae8716f174e43683dc21ca00bfc"
server_key_version: "encryptionkey@0"
I20260926 22:49:45.143936 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "fb237c9d47274161bfa31e7585948461"
format_stamp: "Formatted at 2026-09-26 22:49:45 on dist-test-slave-98hb"
server_key: "ebb932677fdf0fcbe1a9d491ae8e8e65"
server_key_iv: "90f9eae8716f174e43683dc21ca00bfc"
server_key_version: "encryptionkey@0"
I20260926 22:49:45.145078 23932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:49:45.145709 23946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:45.145857 23932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:45.145932 23932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "fb237c9d47274161bfa31e7585948461"
format_stamp: "Formatted at 2026-09-26 22:49:45 on dist-test-slave-98hb"
server_key: "ebb932677fdf0fcbe1a9d491ae8e8e65"
server_key_iv: "90f9eae8716f174e43683dc21ca00bfc"
server_key_version: "encryptionkey@0"
I20260926 22:49:45.145995 23932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:45.149060 23932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:45.149302 23932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:45.149411 23932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:45.149595 23932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:45.149904 23932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:45.149936 23932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:45.149961 23932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:45.149987 23932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:45.155161 23932 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:41199
I20260926 22:49:45.155197 24059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:41199 every 8 connection(s)
I20260926 22:49:45.155480 23932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:45.157210 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 23932
I20260926 22:49:45.157294 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:45.157469 13904 external_mini_cluster.cc:1428] Setting key c193184d55f525e1cb83febb84a4a44f
I20260926 22:49:45.158294 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:45.156142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:37829 (local address 127.13.148.62:36437)
0926 22:49:45.156189 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:45.156191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:45.156389 (+ 198us) server_negotiation.cc:407] Connection header received
0926 22:49:45.156822 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:45.156823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:45.156827 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:45.156877 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:45.158053 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:45.158054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:45.158055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:49:45.158060 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:45.158075 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:49:45.158079 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:45.158128 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:45.158208 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 22:49:45.158224 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:49:45.158470 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:45.156118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36437 (local address 127.13.148.3:37829)
0926 22:49:45.156283 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:45.156314 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:45.156750 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:45.156928 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:45.156930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:45.156938 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:45.157894 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:49:45.157894 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:45.157933 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:49:45.157967 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:45.158144 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:45.158148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:45.158151 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:45.158225 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 22:49:45.158284 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260926 22:49:45.158905 24060 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36437
I20260926 22:49:45.158978 24060 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:45.159087 24060 heartbeater.cc:511] Master 127.13.148.62:36437 requested a full tablet report, sending...
I20260926 22:49:45.159374 23610 ts_manager.cc:194] Registered new tserver with Master: fb237c9d47274161bfa31e7585948461 (127.13.148.3:41199)
I20260926 22:49:45.168996 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 22:49:45.170620 23650 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:49:45.170243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.13.148.62:36437)
0926 22:49:45.170311 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:45.170313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:45.170357 (+ 44us) server_negotiation.cc:407] Connection header received
0926 22:49:45.170417 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:45.170418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:45.170421 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:45.170455 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:45.170560 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52308: BlockingRecv error: recv got EOF from 127.0.0.1:52308 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:49:45.170671 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 23670
I20260926 22:49:45.174541 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 23801
I20260926 22:49:45.178027 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 23932
I20260926 22:49:45.181416 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 23579
2026-09-26T22:49:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (514 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:45 dist-test-slave-98hb krb5kdc[24070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:45 dist-test-slave-98hb krb5kdc[24070](info): set up 2 sockets
Sep 26 22:49:45 dist-test-slave-98hb krb5kdc[24070](info): commencing operation
krb5kdc: starting...
W20260926 22:49:46.126559 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:49:46.236424 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36217
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:46.313825 24083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:46.314015 24083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:46.314034 24083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:46.315146 24083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:49:46.315186 24083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:46.315199 24083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:46.315212 24083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:49:46.315236 24083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:49:46.316717 24083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:46.316942 24083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:46.317137 24083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:46.321558 24089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:46.321645 24088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:46.321660 24091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:46.321681 24083 server_base.cc:1034] running on GCE node
I20260926 22:49:46.322062 24083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:46.322324 24083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:46.323459 24083 hybrid_clock.cc:648] HybridClock initialized: now 1790462986323443 us; error 29 us; skew 500 ppm
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:46.325595 24083 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:49:46.326026 24083 webserver.cc:460] Webserver started at http://127.13.148.62:44145/ using document root <none> and password file <none>
I20260926 22:49:46.326227 24083 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:46.326273 24083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:46.326390 24083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:46.327368 24083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "c042f211e0f949488162c6c771b9fb1c"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "d807368fcb22a69943421fae95a061c3"
server_key_iv: "86f2e5fb43a84857ca6e4c032e0a9dd6"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.327680 24083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "c042f211e0f949488162c6c771b9fb1c"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "d807368fcb22a69943421fae95a061c3"
server_key_iv: "86f2e5fb43a84857ca6e4c032e0a9dd6"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.328853 24083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:46.329464 24098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.329615 24083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:46.329694 24083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "c042f211e0f949488162c6c771b9fb1c"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "d807368fcb22a69943421fae95a061c3"
server_key_iv: "86f2e5fb43a84857ca6e4c032e0a9dd6"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.329780 24083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:46.342855 24083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:46.345012 24083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:46.345141 24083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:46.348742 24083 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36217
I20260926 22:49:46.348779 24150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36217 every 8 connection(s)
I20260926 22:49:46.349068 24083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:49:46.349977 24151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:46.350663 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 24083
I20260926 22:49:46.350759 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:49:46.350943 13904 external_mini_cluster.cc:1428] Setting key f22d1ca5e1088cb369683584bf8a4be9
I20260926 22:49:46.352276 24151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c: Bootstrap starting.
I20260926 22:49:46.352802 24151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:46.353047 24151 log.cc:824] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:46.353633 24151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c: No bootstrap required, opened a new log
I20260926 22:49:46.355326 24151 raft_consensus.cc:348] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c042f211e0f949488162c6c771b9fb1c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36217 } }
I20260926 22:49:46.355523 24151 raft_consensus.cc:374] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:46.355577 24151 raft_consensus.cc:729] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c042f211e0f949488162c6c771b9fb1c, State: Initialized, Role: FOLLOWER
I20260926 22:49:46.355707 24151 consensus_queue.cc:260] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c042f211e0f949488162c6c771b9fb1c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36217 } }
I20260926 22:49:46.355793 24151 raft_consensus.cc:388] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:49:46.355825 24151 raft_consensus.cc:482] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:49:46.355872 24151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:46.356367 24151 raft_consensus.cc:504] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c042f211e0f949488162c6c771b9fb1c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36217 } }
I20260926 22:49:46.356454 24151 leader_election.cc:302] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c042f211e0f949488162c6c771b9fb1c; no voters:
I20260926 22:49:46.356630 24151 leader_election.cc:288] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:49:46.356681 24156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:46.356869 24151 sys_catalog.cc:564] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:49:46.356875 24156 raft_consensus.cc:686] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [term 1 LEADER]: Becoming Leader. State: Replica: c042f211e0f949488162c6c771b9fb1c, State: Running, Role: LEADER
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:46.357045 24156 consensus_queue.cc:237] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c042f211e0f949488162c6c771b9fb1c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36217 } }
I20260926 22:49:46.357573 24158 sys_catalog.cc:455] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c042f211e0f949488162c6c771b9fb1c. Latest consensus state: current_term: 1 leader_uuid: "c042f211e0f949488162c6c771b9fb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c042f211e0f949488162c6c771b9fb1c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36217 } } }
I20260926 22:49:46.357650 24158 sys_catalog.cc:458] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:46.357849 24157 sys_catalog.cc:455] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c042f211e0f949488162c6c771b9fb1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c042f211e0f949488162c6c771b9fb1c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36217 } } }
I20260926 22:49:46.357920 24157 sys_catalog.cc:458] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c [sys.catalog]: This master's current role is: LEADER
I20260926 22:49:46.358726 24162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:49:46.359359 24162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:49:46.361142 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.351393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.13.148.62:36217)
0926 22:49:46.351562 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.351572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.351598 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:49:46.352123 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.352130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.352175 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.352303 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:46.352877 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.353876 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.354570 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.354969 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.355061 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.357631 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:46.357635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:46.357635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:46.357644 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:46.359543 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.359934 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.359937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.359937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.360037 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.360238 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.360239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.360239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.360466 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:46.360589 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.360819 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.360949 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260926 22:49:46.361474 24162 catalog_manager.cc:1347] Generated new cluster ID: d4510269205a4ae6b546fc32c6dc6ed9
I20260926 22:49:46.361516 24162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:49:46.368041 24162 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:49:46.368795 24162 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:49:46.375299 24162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c: Generated new TSK 0
I20260926 22:49:46.375465 24162 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:49:46.402038 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36217
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:46.479627 24177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:46.479825 24177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:46.479846 24177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:46.480921 24177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:46.480953 24177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:46.480978 24177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:49:46.482384 24177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:46.482564 24177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:46.482736 24177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:46.487283 24182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:46.487294 24185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:46.487389 24177 server_base.cc:1034] running on GCE node
W20260926 22:49:46.487331 24183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:46.487710 24177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:46.487946 24177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:46.489081 24177 hybrid_clock.cc:648] HybridClock initialized: now 1790462986489068 us; error 28 us; skew 500 ppm
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:46.491298 24177 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:49:46.491678 24177 webserver.cc:460] Webserver started at http://127.13.148.1:45399/ using document root <none> and password file <none>
I20260926 22:49:46.491883 24177 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:46.491933 24177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:46.492054 24177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:46.493031 24177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "fd97efe1aa884c4bbd71021a55c1ea25"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "b63fc194fc70feb26823b5d94b5b0b10"
server_key_iv: "0ac53720bf449f0287e26fc6be7d1cb6"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.493383 24177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "fd97efe1aa884c4bbd71021a55c1ea25"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "b63fc194fc70feb26823b5d94b5b0b10"
server_key_iv: "0ac53720bf449f0287e26fc6be7d1cb6"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.494676 24177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:49:46.495309 24192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.495455 24177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:49:46.495532 24177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "fd97efe1aa884c4bbd71021a55c1ea25"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "b63fc194fc70feb26823b5d94b5b0b10"
server_key_iv: "0ac53720bf449f0287e26fc6be7d1cb6"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.495592 24177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:46.516486 24177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:46.518213 24177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:46.518353 24177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:46.518545 24177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:46.518787 24177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:46.518816 24177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.518836 24177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:46.518852 24177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.524070 24177 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:34971
I20260926 22:49:46.524132 24305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:34971 every 8 connection(s)
I20260926 22:49:46.524406 24177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:49:46.526361 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 24177
I20260926 22:49:46.526448 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:49:46.526612 13904 external_mini_cluster.cc:1428] Setting key 9c15ebbed65ad49842099ff36171213a
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:46.533475 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.525060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:58231 (local address 127.13.148.62:36217)
0926 22:49:46.525123 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.525125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.525298 (+ 173us) server_negotiation.cc:407] Connection header received
0926 22:49:46.525807 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.525809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.525821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.525871 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:46.526864 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.527524 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.528553 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.528755 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.528806 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.530964 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:46.530966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:46.530966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:46.530973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:46.532207 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.532579 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.532580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.532581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.532638 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.532851 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.532852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.532852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.532961 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:46.533016 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.533265 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.533383 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:49:46.533599 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.525030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36217 (local address 127.13.148.1:58231)
0926 22:49:46.525174 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.525220 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:46.525690 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:46.525939 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:46.525943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:46.526533 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:46.526760 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.526767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.527608 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.527608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.528470 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.528473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.528827 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.528828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.528962 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.529424 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:46.529433 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:46.530878 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:46.532302 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:46.532309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:46.532311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:46.532509 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:46.532695 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:46.532695 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:46.532696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:46.532793 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:46.533017 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:46.533023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:46.533141 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 22:49:46.533236 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 22:49:46.533393 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260926 22:49:46.534044 24306 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36217
I20260926 22:49:46.534133 24306 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:46.534274 24306 heartbeater.cc:511] Master 127.13.148.62:36217 requested a full tablet report, sending...
I20260926 22:49:46.534725 24115 ts_manager.cc:194] Registered new tserver with Master: fd97efe1aa884c4bbd71021a55c1ea25 (127.13.148.1:34971)
I20260926 22:49:46.535372 24115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:58231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:49:46.548321 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36217
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:46.625936 24311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:46.626127 24311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:46.626147 24311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:46.627249 24311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:46.627286 24311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:46.627310 24311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:49:46.628656 24311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:46.628834 24311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:46.629005 24311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:49:46.633659 24311 server_base.cc:1034] running on GCE node
W20260926 22:49:46.633669 24317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:46.633670 24316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:46.633805 24319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:46.634076 24311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:46.634310 24311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:46.635435 24311 hybrid_clock.cc:648] HybridClock initialized: now 1790462986635420 us; error 28 us; skew 500 ppm
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:46.637617 24311 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:49:46.638062 24311 webserver.cc:460] Webserver started at http://127.13.148.2:32947/ using document root <none> and password file <none>
I20260926 22:49:46.638260 24311 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:46.638306 24311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:46.638427 24311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:46.639400 24311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "4e581829be5a4900b7490512ace7e31c"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "35104b0fd980bceba7ba120a71b7d418"
server_key_iv: "14a2b36eb92b6703cd6b4dcea641662d"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.639714 24311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "4e581829be5a4900b7490512ace7e31c"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "35104b0fd980bceba7ba120a71b7d418"
server_key_iv: "14a2b36eb92b6703cd6b4dcea641662d"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.640885 24311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:49:46.641583 24326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.641781 24311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:46.641866 24311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "4e581829be5a4900b7490512ace7e31c"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "35104b0fd980bceba7ba120a71b7d418"
server_key_iv: "14a2b36eb92b6703cd6b4dcea641662d"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.641932 24311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:46.684180 24311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:46.686018 24311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:46.686148 24311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:46.686370 24311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:46.686655 24311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:46.686687 24311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.686717 24311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:46.686744 24311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.692095 24311 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:40573
I20260926 22:49:46.692149 24439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:40573 every 8 connection(s)
I20260926 22:49:46.692435 24311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:49:46.692741 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 24311
I20260926 22:49:46.692847 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:49:46.693070 13904 external_mini_cluster.cc:1428] Setting key 1f3a6125f3aa96c18d9038205b9dfe32
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260926 22:49:46.702106 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.693084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:47549 (local address 127.13.148.62:36217)
0926 22:49:46.693135 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.693138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.693413 (+ 275us) server_negotiation.cc:407] Connection header received
0926 22:49:46.694052 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.694054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.694064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.694121 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:46.695063 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.695546 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.696895 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.697089 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.697149 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.699456 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:46.699458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:46.699458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:46.699466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:46.700734 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.701204 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.701205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.701205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.701257 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.701473 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.701475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.701475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.701579 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:46.701634 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.701938 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.702017 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:46.702275 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.693057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36217 (local address 127.13.148.2:47549)
0926 22:49:46.693296 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.693344 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:46.693958 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:46.694185 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:46.694188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:46.694733 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:46.694962 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.694969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.695625 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.695627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.696818 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.696821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.697173 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.697174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.697326 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.697806 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:46.697816 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:46.699371 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:46.700807 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:46.700813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:46.700816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:46.701119 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:46.701313 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:46.701314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:46.701314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:46.701416 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:46.701636 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:46.701642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:46.701812 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 22:49:46.701911 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 22:49:46.702066 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":73,"threads_started":1}
I20260926 22:49:46.702656 24440 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36217
I20260926 22:49:46.702780 24440 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:46.702976 24440 heartbeater.cc:511] Master 127.13.148.62:36217 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
I20260926 22:49:46.703429 24115 ts_manager.cc:194] Registered new tserver with Master: 4e581829be5a4900b7490512ace7e31c (127.13.148.2:40573)
I20260926 22:49:46.703778 24115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:47549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:49:46.716760 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36217
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:49:46.795857 24445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:49:46.796087 24445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:49:46.796109 24445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:49:46.797290 24445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:49:46.797331 24445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:49:46.797356 24445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:49:46.798905 24445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:36217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:49:46.799134 24445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:49:46.799369 24445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:49:46.804436 24450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:46.804435 24451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:49:46.804454 24453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:49:46.804575 24445 server_base.cc:1034] running on GCE node
I20260926 22:49:46.804837 24445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:49:46.805054 24445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:49:46.806203 24445 hybrid_clock.cc:648] HybridClock initialized: now 1790462986806186 us; error 29 us; skew 500 ppm
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:49:46.808517 24445 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:49:46.808964 24445 webserver.cc:460] Webserver started at http://127.13.148.3:46859/ using document root <none> and password file <none>
I20260926 22:49:46.809165 24445 fs_manager.cc:359] Metadata directory not provided
I20260926 22:49:46.809211 24445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:49:46.809332 24445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:49:46.810379 24445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "f538dfb9b9c547a3b948a411c16a7de2"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "3190d245dcb7714b0eb4e7df972f0c61"
server_key_iv: "023047f7c6c0f31f29641275ba8983f5"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.810699 24445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "f538dfb9b9c547a3b948a411c16a7de2"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "3190d245dcb7714b0eb4e7df972f0c61"
server_key_iv: "023047f7c6c0f31f29641275ba8983f5"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.811961 24445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:49:46.812732 24460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.812950 24445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:49:46.813043 24445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "f538dfb9b9c547a3b948a411c16a7de2"
format_stamp: "Formatted at 2026-09-26 22:49:46 on dist-test-slave-98hb"
server_key: "3190d245dcb7714b0eb4e7df972f0c61"
server_key_iv: "023047f7c6c0f31f29641275ba8983f5"
server_key_version: "encryptionkey@0"
I20260926 22:49:46.813118 24445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:49:46.828192 24445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:49:46.830379 24445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:49:46.830513 24445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:49:46.830741 24445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:49:46.831044 24445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:49:46.831077 24445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.831105 24445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:49:46.831123 24445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:49:46.836577 24445 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:45795
I20260926 22:49:46.836634 24573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:45795 every 8 connection(s)
I20260926 22:49:46.836949 24445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:49:46.840942 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 24445
I20260926 22:49:46.841051 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:49:46.841264 13904 external_mini_cluster.cc:1428] Setting key 1bbaf86ff69d5b61249ecdf5bd05264b
Sep 26 22:49:46 dist-test-slave-98hb krb5kdc[24070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462986, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:49:46.845887 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.837640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:36383 (local address 127.13.148.62:36217)
0926 22:49:46.837725 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.837746 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.837915 (+ 169us) server_negotiation.cc:407] Connection header received
0926 22:49:46.838411 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.838412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.838423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.838475 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:46.839394 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.839884 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.840818 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.841127 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.841192 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.843387 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:46.843388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:46.843388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:46.843395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:46.844688 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.845041 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.845041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.845041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.845087 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.845274 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.845275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.845275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.845377 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:46.845435 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.845704 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.845800 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 22:49:46.846062 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.837625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36217 (local address 127.13.148.3:36383)
0926 22:49:46.837792 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.837835 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:46.838326 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:46.838550 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:46.838553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:46.839089 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:49:46.839292 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.839296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.839949 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.839950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.840735 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.840737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.841212 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.841213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.841319 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.841872 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:49:46.841882 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:49:46.843296 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:49:46.844758 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:46.844764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:46.844766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:46.844979 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:46.845133 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:49:46.845133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:49:46.845134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:49:46.845231 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:49:46.845437 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:49:46.845443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:49:46.845578 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 22:49:46.845668 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 22:49:46.845850 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260926 22:49:46.846513 24574 heartbeater.cc:347] Connected to a master server at 127.13.148.62:36217
I20260926 22:49:46.846597 24574 heartbeater.cc:464] Registering TS with master...
I20260926 22:49:46.846740 24574 heartbeater.cc:511] Master 127.13.148.62:36217 requested a full tablet report, sending...
I20260926 22:49:46.847060 24115 ts_manager.cc:194] Registered new tserver with Master: f538dfb9b9c547a3b948a411c16a7de2 (127.13.148.3:45795)
I20260926 22:49:46.847405 24115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:36383
I20260926 22:49:46.852993 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:49:46.859594 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.854241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.13.148.62:36217)
0926 22:49:46.854291 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.854293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.854405 (+ 112us) server_negotiation.cc:407] Connection header received
0926 22:49:46.854445 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.854445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.854453 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.854497 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:49:46.854984 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.855442 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.856101 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.856281 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.856335 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.857275 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:49:46.857276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:49:46.857276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:49:46.857281 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:49:46.858702 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.858979 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.858980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.858980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.859023 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:49:46.859160 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:49:46.859161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:49:46.859161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:49:46.859327 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:49:46.859398 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.859481 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.859535 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:49:46.860863 24115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:49:46.862457 24115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:49:46.868593 24582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.864308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45795 (local address 127.0.0.1:34800)
0926 22:49:46.864413 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.864452 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:46.864513 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:46.864839 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:46.864842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:46.864844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:46.864911 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.864912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.866213 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.866221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.867411 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.867414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.868328 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.868329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.868392 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.868395 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:46.868437 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 22:49:46.868489 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260926 22:49:46.868706 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.864319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34971 (local address 127.0.0.1:37204)
0926 22:49:46.864829 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.864838 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:46.864876 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:46.865117 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:46.865118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:46.865120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:46.865158 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.865160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.866074 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.866075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.867411 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.867414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.868303 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.868304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.868378 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.868394 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 22:49:46.868463 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 22:49:46.868505 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":76,"thread_start_us":57,"threads_started":1}
I20260926 22:49:46.868741 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.864472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.13.148.3:45795)
0926 22:49:46.864657 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.864662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.864673 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:49:46.864730 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.864732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.864749 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.864825 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:46.864988 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.866114 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.867498 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.868241 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.868550 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.868609 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.868631 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.868668 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260926 22:49:46.868604 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.864538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.13.148.1:34971)
0926 22:49:46.864695 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.864700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.864927 (+ 227us) server_negotiation.cc:407] Connection header received
0926 22:49:46.864966 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.864969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.864982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.865045 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:46.865237 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.865987 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.867498 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.868223 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.868305 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.868366 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.868494 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.868534 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260926 22:49:46.869437 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.864313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:40573 (local address 127.0.0.1:50778)
0926 22:49:46.864690 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.864700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:46.864747 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:46.864915 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:46.864916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:46.864917 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:46.864972 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.864973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.867500 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.867501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.868148 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.868149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.869244 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.869245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.869292 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.869295 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:46.869341 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:49:46.869376 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":55,"threads_started":1}
I20260926 22:49:46.869473 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.864549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.13.148.2:40573)
0926 22:49:46.864730 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.864734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.864743 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:49:46.864824 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.864826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.864841 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.864919 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:46.865050 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.866130 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.868229 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.869097 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.869160 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.869213 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.869364 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.869403 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260926 22:49:46.869419 24508 tablet_service.cc:1467] Processing CreateTablet for tablet 77decf6220d4434aa5cee5d16127a003 (DEFAULT_TABLE table=test-table [id=fea50b72ebb44049b5cfe94f483e21c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:46.869779 24508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77decf6220d4434aa5cee5d16127a003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:46.869715 24240 tablet_service.cc:1467] Processing CreateTablet for tablet 77decf6220d4434aa5cee5d16127a003 (DEFAULT_TABLE table=test-table [id=fea50b72ebb44049b5cfe94f483e21c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:46.870046 24240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77decf6220d4434aa5cee5d16127a003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:46.870172 24374 tablet_service.cc:1467] Processing CreateTablet for tablet 77decf6220d4434aa5cee5d16127a003 (DEFAULT_TABLE table=test-table [id=fea50b72ebb44049b5cfe94f483e21c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:49:46.870442 24374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77decf6220d4434aa5cee5d16127a003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:49:46.872692 24588 tablet_bootstrap.cc:492] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Bootstrap starting.
I20260926 22:49:46.872972 24590 tablet_bootstrap.cc:492] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25: Bootstrap starting.
I20260926 22:49:46.873474 24588 tablet_bootstrap.cc:654] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:46.873529 24590 tablet_bootstrap.cc:654] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:46.873875 24589 tablet_bootstrap.cc:492] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: Bootstrap starting.
I20260926 22:49:46.874060 24588 log.cc:824] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:46.874061 24590 log.cc:824] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:46.874428 24589 tablet_bootstrap.cc:654] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: Neither blocks nor log segments found. Creating new log.
I20260926 22:49:46.874805 24588 tablet_bootstrap.cc:492] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: No bootstrap required, opened a new log
I20260926 22:49:46.874873 24588 ts_tablet_manager.cc:1397] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:46.874986 24590 tablet_bootstrap.cc:492] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25: No bootstrap required, opened a new log
I20260926 22:49:46.874984 24589 log.cc:824] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: Log is configured to *not* fsync() on all Append() calls
I20260926 22:49:46.875048 24590 ts_tablet_manager.cc:1397] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:49:46.875581 24589 tablet_bootstrap.cc:492] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: No bootstrap required, opened a new log
I20260926 22:49:46.875638 24589 ts_tablet_manager.cc:1397] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:49:46.876796 24588 raft_consensus.cc:348] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.876966 24588 raft_consensus.cc:374] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:46.877003 24588 raft_consensus.cc:729] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e581829be5a4900b7490512ace7e31c, State: Initialized, Role: FOLLOWER
I20260926 22:49:46.877017 24590 raft_consensus.cc:348] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.877048 24589 raft_consensus.cc:348] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.877177 24590 raft_consensus.cc:374] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:46.877146 24588 consensus_queue.cc:260] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.877223 24590 raft_consensus.cc:729] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd97efe1aa884c4bbd71021a55c1ea25, State: Initialized, Role: FOLLOWER
I20260926 22:49:46.877233 24589 raft_consensus.cc:374] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:49:46.877272 24589 raft_consensus.cc:729] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f538dfb9b9c547a3b948a411c16a7de2, State: Initialized, Role: FOLLOWER
I20260926 22:49:46.877404 24588 ts_tablet_manager.cc:1428] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:46.877369 24590 consensus_queue.cc:260] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.877424 24589 consensus_queue.cc:260] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.877612 24440 heartbeater.cc:503] Master 127.13.148.62:36217 was elected leader, sending a full tablet report...
I20260926 22:49:46.877606 24590 ts_tablet_manager.cc:1428] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 22:49:46.877619 24306 heartbeater.cc:503] Master 127.13.148.62:36217 was elected leader, sending a full tablet report...
I20260926 22:49:46.877677 24589 ts_tablet_manager.cc:1428] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:49:46.877805 24574 heartbeater.cc:503] Master 127.13.148.62:36217 was elected leader, sending a full tablet report...
I20260926 22:49:46.934844 24595 raft_consensus.cc:482] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:49:46.934962 24595 raft_consensus.cc:504] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.935304 24595 leader_election.cc:288] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f538dfb9b9c547a3b948a411c16a7de2 (127.13.148.3:45795), 4e581829be5a4900b7490512ace7e31c (127.13.148.2:40573)
I20260926 22:49:46.938060 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.935532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:44313 (local address 127.13.148.2:40573)
0926 22:49:46.935576 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.935578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.935724 (+ 146us) server_negotiation.cc:407] Connection header received
0926 22:49:46.935754 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.935755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.935765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.935814 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:46.935954 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.936414 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.937368 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.937761 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.937827 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.937836 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.937961 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.937997 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:49:46.938078 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.935574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:35405 (local address 127.13.148.3:45795)
0926 22:49:46.935617 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.935618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.935626 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:49:46.935650 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.935650 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.935659 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.935703 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:49:46.935864 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.936369 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.937369 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.937764 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.937831 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.937840 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.937976 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.938012 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:49:46.938091 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.935465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45795 (local address 127.13.148.1:35405)
0926 22:49:46.935528 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.935540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:46.935585 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:46.935727 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:46.935728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:46.935729 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:46.935799 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.935801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.936449 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.936450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.937277 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.937279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.937834 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.937835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.937913 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.937916 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:49:46.937960 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:49:46.937998 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":51}
I20260926 22:49:46.938097 24597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.935478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:40573 (local address 127.13.148.1:44313)
0926 22:49:46.935643 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.935653 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:49:46.935710 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:49:46.935826 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:49:46.935827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:49:46.935828 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:49:46.935892 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.935894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.936492 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.936493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.937276 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:49:46.937278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.937829 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.937830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:49:46.937895 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.937899 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:49:46.937946 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:49:46.937982 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":120,"threads_started":1}
I20260926 22:49:46.938416 24528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77decf6220d4434aa5cee5d16127a003" candidate_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f538dfb9b9c547a3b948a411c16a7de2" is_pre_election: true
I20260926 22:49:46.938445 24394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77decf6220d4434aa5cee5d16127a003" candidate_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e581829be5a4900b7490512ace7e31c" is_pre_election: true
I20260926 22:49:46.938568 24394 raft_consensus.cc:2454] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd97efe1aa884c4bbd71021a55c1ea25 in term 0.
I20260926 22:49:46.938575 24528 raft_consensus.cc:2454] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd97efe1aa884c4bbd71021a55c1ea25 in term 0.
I20260926 22:49:46.938784 24196 leader_election.cc:302] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e581829be5a4900b7490512ace7e31c, fd97efe1aa884c4bbd71021a55c1ea25; no voters:
I20260926 22:49:46.938947 24595 raft_consensus.cc:2781] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:49:46.938984 24595 raft_consensus.cc:482] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:49:46.938998 24595 raft_consensus.cc:3037] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:46.939726 24595 raft_consensus.cc:504] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.939852 24595 leader_election.cc:288] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [CANDIDATE]: Term 1 election: Requested vote from peers f538dfb9b9c547a3b948a411c16a7de2 (127.13.148.3:45795), 4e581829be5a4900b7490512ace7e31c (127.13.148.2:40573)
I20260926 22:49:46.940004 24394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77decf6220d4434aa5cee5d16127a003" candidate_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e581829be5a4900b7490512ace7e31c"
I20260926 22:49:46.940069 24394 raft_consensus.cc:3037] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:46.940667 24394 raft_consensus.cc:2454] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd97efe1aa884c4bbd71021a55c1ea25 in term 1.
I20260926 22:49:46.940759 24528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77decf6220d4434aa5cee5d16127a003" candidate_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f538dfb9b9c547a3b948a411c16a7de2"
I20260926 22:49:46.940833 24196 leader_election.cc:302] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e581829be5a4900b7490512ace7e31c, fd97efe1aa884c4bbd71021a55c1ea25; no voters:
I20260926 22:49:46.940835 24528 raft_consensus.cc:3037] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:49:46.940930 24595 raft_consensus.cc:2781] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:49:46.941083 24595 raft_consensus.cc:686] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 1 LEADER]: Becoming Leader. State: Replica: fd97efe1aa884c4bbd71021a55c1ea25, State: Running, Role: LEADER
I20260926 22:49:46.941171 24595 consensus_queue.cc:237] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } }
I20260926 22:49:46.941653 24528 raft_consensus.cc:2454] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd97efe1aa884c4bbd71021a55c1ea25 in term 1.
I20260926 22:49:46.941964 24115 catalog_manager.cc:5495] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd97efe1aa884c4bbd71021a55c1ea25 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd97efe1aa884c4bbd71021a55c1ea25" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34971 } health_report { overall_health: HEALTHY } } }
W20260926 22:49:46.942900 24441 tablet.cc:2369] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:49:46.962200 24586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.960052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.13.148.1:34971)
0926 22:49:46.960117 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:49:46.960118 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:49:46.960125 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:49:46.960157 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:49:46.960158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:49:46.960175 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:49:46.960234 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:49:46.960347 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.960807 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.961494 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:49:46.961694 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:49:46.961794 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:49:46.961904 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:49:46.962006 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:49:46.962063 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:49:46.962082 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 22:49:46.962100 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 22:49:46.963912 24394 raft_consensus.cc:1261] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 1 FOLLOWER]: Refusing update from remote peer fd97efe1aa884c4bbd71021a55c1ea25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:46.963919 24528 raft_consensus.cc:1261] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 1 FOLLOWER]: Refusing update from remote peer fd97efe1aa884c4bbd71021a55c1ea25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:49:46.964243 24595 consensus_queue.cc:1035] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e581829be5a4900b7490512ace7e31c" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:46.964389 24598 consensus_queue.cc:1035] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f538dfb9b9c547a3b948a411c16a7de2" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:49:46.966790 24603 mvcc.cc:205] Tried to move back new op lower bound from 7333736394601652224 to 7333736394511544320. Current Snapshot: MvccSnapshot[applied={T|T < 7333736394601652224}]
I20260926 22:49:46.967101 24602 mvcc.cc:205] Tried to move back new op lower bound from 7333736394601652224 to 7333736394511544320. Current Snapshot: MvccSnapshot[applied={T|T < 7333736394601652224}]
I20260926 22:49:46.968958 24114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:46.969043 24114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:49:46.969763 24114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c042f211e0f949488162c6c771b9fb1c: Sending DeleteTablet for 3 replicas of tablet 77decf6220d4434aa5cee5d16127a003
I20260926 22:49:46.970074 24240 tablet_service.cc:1514] Processing DeleteTablet for tablet 77decf6220d4434aa5cee5d16127a003 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:46 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:37204
I20260926 22:49:46.970067 24374 tablet_service.cc:1514] Processing DeleteTablet for tablet 77decf6220d4434aa5cee5d16127a003 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:46 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:50778
I20260926 22:49:46.970091 24508 tablet_service.cc:1514] Processing DeleteTablet for tablet 77decf6220d4434aa5cee5d16127a003 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:49:46 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:34800
I20260926 22:49:46.970295 24613 tablet_replica.cc:330] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: stopping tablet replica
I20260926 22:49:46.970302 24611 tablet_replica.cc:330] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25: stopping tablet replica
I20260926 22:49:46.970392 24611 raft_consensus.cc:2229] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:49:46.970412 24613 raft_consensus.cc:2229] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:46.970535 24613 raft_consensus.cc:2258] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:46.970528 24611 raft_consensus.cc:2258] T 77decf6220d4434aa5cee5d16127a003 P fd97efe1aa884c4bbd71021a55c1ea25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:46.970579 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 24177
I20260926 22:49:46.971347 24613 ts_tablet_manager.cc:1905] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:46.972191 24612 tablet_replica.cc:330] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: stopping tablet replica
I20260926 22:49:46.972294 24612 raft_consensus.cc:2229] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:49:46.972393 24612 raft_consensus.cc:2258] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:49:46.972801 24612 ts_tablet_manager.cc:1905] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:49:46.973001 24613 ts_tablet_manager.cc:1918] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:46.973065 24613 log.cc:1196] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/77decf6220d4434aa5cee5d16127a003
I20260926 22:49:46.973358 24613 ts_tablet_manager.cc:1939] T 77decf6220d4434aa5cee5d16127a003 P 4e581829be5a4900b7490512ace7e31c: Deleting consensus metadata
I20260926 22:49:46.973783 24102 catalog_manager.cc:4841] TS 4e581829be5a4900b7490512ace7e31c (127.13.148.2:40573): tablet 77decf6220d4434aa5cee5d16127a003 (table test-table [id=fea50b72ebb44049b5cfe94f483e21c1]) successfully deleted
I20260926 22:49:46.974159 24612 ts_tablet_manager.cc:1918] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:49:46.974215 24612 log.cc:1196] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/77decf6220d4434aa5cee5d16127a003
I20260926 22:49:46.974380 24612 ts_tablet_manager.cc:1939] T 77decf6220d4434aa5cee5d16127a003 P f538dfb9b9c547a3b948a411c16a7de2: Deleting consensus metadata
I20260926 22:49:46.974790 24099 catalog_manager.cc:4841] TS f538dfb9b9c547a3b948a411c16a7de2 (127.13.148.3:45795): tablet 77decf6220d4434aa5cee5d16127a003 (table test-table [id=fea50b72ebb44049b5cfe94f483e21c1]) successfully deleted
W20260926 22:49:46.976011 24101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:34971 (error 108)
I20260926 22:49:46.976176 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 24311
I20260926 22:49:46.976881 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:49:46.976758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34971 (local address 127.0.0.1:37218)
0926 22:49:46.976813 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:49:46.976850 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 22:49:46.976979 24101 catalog_manager.cc:4584] TS fd97efe1aa884c4bbd71021a55c1ea25 (127.13.148.1:34971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 77decf6220d4434aa5cee5d16127a003: Network error: Client connection negotiation failed: client connection to 127.13.148.1:34971: connect: Connection refused (error 111)
I20260926 22:49:46.980373 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 24445
I20260926 22:49:46.984079 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 24083
2026-09-26T22:49:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1803 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:49:47 dist-test-slave-98hb krb5kdc[24617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:49:47 dist-test-slave-98hb krb5kdc[24617](info): set up 2 sockets
Sep 26 22:49:47 dist-test-slave-98hb krb5kdc[24617](info): commencing operation
krb5kdc: starting...
W20260926 22:49:47.934639 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:49:48 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790462988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:49:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/rangeradmin_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/rangeradmin_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/rangerlookup_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/rangerlookup_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
I20260926 22:49:48.087935 13904 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskClAdwy/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/postgres -l logfile start
2026-09-26 22:49:50.392 UTC [24647] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 22:49:50.392 UTC [24647] LOG: listening on IPv4 address "127.13.148.20", port 45635
2026-09-26 22:49:50.395 UTC [24647] LOG: listening on Unix socket "/tmp/.s.PGSQL.45635"
2026-09-26 22:49:50.414 UTC [24650] LOG: database system was shut down at 2026-09-26 22:49:49 UTC
2026-09-26 22:49:50.417 UTC [24647] LOG: database system is ready to accept connections
I20260926 22:49:51.282548 13904 mini_postgres.cc:96] Postgres bound to 45635
2026-09-26 22:49:51.288 UTC [24658] FATAL: database "slave" does not exist
127.13.148.20:45635 - accepting connections
I20260926 22:49:51.289299 13904 mini_ranger.cc:161] Starting Ranger...
I20260926 22:49:51.302964 13904 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 22:49:51.748394 13904 mini_ranger.cc:87] Created ranger Postgres database
I20260926 22:49:51.748461 13904 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-admin
2026-09-26 22:49:52,023 [I] DB FLAVOR :POSTGRES
2026-09-26 22:49:52,024 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 22:49:52,024 [I] Checking connection..
2026-09-26 22:49:52,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 22:49:52,272 [I] Checking connection passed.
2026-09-26 22:49:52,272 [I] --------- Verifying version history table ---------
2026-09-26 22:49:52,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:49:52,521 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 22:49:52,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:49:52,763 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 22:49:52,763 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 22:49:52,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 22:49:53.005 UTC [24722] WARNING: there is no transaction in progress
2026-09-26 22:49:53,010 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 22:49:53,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:49:53,242 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 22:49:53,243 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 22:49:53,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 22:49:53,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 22:49:53,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-98hb.c.gcp-upstream.internal','N') ;"
2026-09-26 22:49:53,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 22:49:54,209 [I] Table x_portal_user does not exist in database ranger
2026-09-26 22:49:54,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 22:49:54,429 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 22:49:54,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 22:49:54,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 22:49:54,915 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 22:49:54,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 22:49:55.816 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.831 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.847 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.860 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.878 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.897 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.948 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.954 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.967 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.976 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.986 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:55.999 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:56.008 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:56.015 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:56.021 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:56.027 UTC [24830] WARNING: there is no transaction in progress
2026-09-26 22:49:56,582 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 22:49:56,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 22:49:56,823 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 22:49:56,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 22:49:57,070 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 22:49:57,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 22:49:57,311 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 22:49:57,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 22:49:57,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 22:49:57,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-98hb.c.gcp-upstream.internal';"
2026-09-26 22:49:58,025 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 22:49:58,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 22:49:58,271 [I] DB_PATCHES have already been applied
I20260926 22:49:58.276912 13904 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-admin:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskClAdwy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 22:49:58.276974 13904 mini_ranger.cc:193] Using host: 127.13.148.20
I20260926 22:49:58.279822 13904 mini_ranger.cc:227] Ranger admin URL: http://127.13.148.20:45117
Sep 26, 2026 10:49:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 10:49:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 10:49:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 10:49:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 10:49:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 10:49:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 10:49:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 22:49:59.291688 13904 mini_ranger.cc:160] Time spent starting Ranger: real 8.002s user 0.000s sys 0.008s
Sep 26, 2026 10:50:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 10:50:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 10:50:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 10:50:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 10:50:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 22:50:19.381891 13904 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/rangerkms_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/rangerkms_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.148.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 22:50:19 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463019, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 22:50:19.471992 13904 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 22:50:19.480177 13904 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 22:50:20.222918 13904 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 22:50:20.222981 13904 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms
I20260926 22:50:20.222992 13904 mini_ranger_kms.cc:227] Using postgres at 127.13.148.20:45635
2026-09-26 22:50:20,412 [I] DB FLAVOR :POSTGRES
2026-09-26 22:50:20,412 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 22:50:20,412 [I] Checking connection
2026-09-26 22:50:20,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 22:50:20,632 [I] connection success
2026-09-26 22:50:20,632 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 22:50:20,632 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 22:50:20,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 22:50:20,877 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 22:50:20,878 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 22:50:20,878 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 22:50:20,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:45635/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 22:50:21,144 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 22:50:21.597779 13904 mini_ranger_kms.cc:310] Created kms service
I20260926 22:50:21.783774 13904 mini_ranger_kms.cc:326] Created kudu user
I20260926 22:50:21.824990 13904 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 22:50:21 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463019, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:50:22.202188 13904 mini_ranger_kms.cc:384] Added ranger policy
I20260926 22:50:22.202307 13904 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms:/tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 22:50:22.202329 13904 mini_ranger_kms.cc:240] Using host: 127.13.148.20
I20260926 22:50:22.205866 13904 mini_ranger_kms.cc:276] Ranger KMS PID: 25010
I20260926 22:50:22.205994 13904 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.148.20:38277
Sep 26, 2026 10:50:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 10:50:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 10:50:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 10:50:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 10:50:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 10:50:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 22:50:24.233987 13904 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.762s user 0.014s sys 0.159s
I20260926 22:50:24.234098 13904 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 22:50:24.234184 13904 mini_ranger_kms.cc:401] 127.13.148.20:38277/kms/v1/keys
Sep 26 22:50:24 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790463024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 22:50:24 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Sep 26 22:50:25 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463024, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Sep 26 22:50:27 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:50:27.346911 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40591
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40591
--ranger_config_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:38277/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:50:27.424825 25061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:50:27.425019 25061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:50:27.425038 25061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:50:27.426178 25061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:50:27.426216 25061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:50:27.426229 25061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:50:27.426239 25061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:50:27.426249 25061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:50:27.427719 25061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.148.20:38277/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40591
--ranger_config_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:50:27.427928 25061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:50:27.428138 25061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:50:27.432631 25066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:50:27.432646 25069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:50:27.432806 25061 server_base.cc:1034] running on GCE node
W20260926 22:50:27.432618 25067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:50:27.433089 25061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:50:27.433374 25061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:50:27.434505 25061 hybrid_clock.cc:648] HybridClock initialized: now 1790463027434491 us; error 31 us; skew 500 ppm
Sep 26 22:50:27 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463027, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:50:27.442745 25061 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:50:27.443277 25061 webserver.cc:460] Webserver started at http://127.13.148.62:38341/ using document root <none> and password file <none>
I20260926 22:50:27.443501 25061 fs_manager.cc:359] Metadata directory not provided
I20260926 22:50:27.443548 25061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:50:27.443681 25061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:50:27 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463027, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:50:27.488722 25061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "c37aa0daf0f949c8baaf0cb1659f78d5"
format_stamp: "Formatted at 2026-09-26 22:50:27 on dist-test-slave-98hb"
server_key: "2d7061ea982234a34b396c90410ac2a6"
server_key_iv: "10083ca7b9215f5083501b007f7d3d73"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:27.489159 25061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "c37aa0daf0f949c8baaf0cb1659f78d5"
format_stamp: "Formatted at 2026-09-26 22:50:27 on dist-test-slave-98hb"
server_key: "2d7061ea982234a34b396c90410ac2a6"
server_key_iv: "10083ca7b9215f5083501b007f7d3d73"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:27.490809 25061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:50:27.504323 25076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:27.504532 25061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:50:27.504618 25061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "c37aa0daf0f949c8baaf0cb1659f78d5"
format_stamp: "Formatted at 2026-09-26 22:50:27 on dist-test-slave-98hb"
server_key: "2d7061ea982234a34b396c90410ac2a6"
server_key_iv: "10083ca7b9215f5083501b007f7d3d73"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:27.504724 25061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:50:27.514726 25061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:50:27.516523 25061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:50:27.516659 25061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:50:27.521353 25061 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40591
I20260926 22:50:27.521378 25128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40591 every 8 connection(s)
I20260926 22:50:27.521802 25061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:50:27.522492 25129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:50:27.522552 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 25061
I20260926 22:50:27.522646 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:50:27.522843 13904 external_mini_cluster.cc:1428] Setting key 075a4bc0b2081e89611346ba6b20e88c
I20260926 22:50:27.525094 25129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5: Bootstrap starting.
I20260926 22:50:27.525646 25129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5: Neither blocks nor log segments found. Creating new log.
I20260926 22:50:27.526058 25129 log.cc:824] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5: Log is configured to *not* fsync() on all Append() calls
I20260926 22:50:27.526733 25129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5: No bootstrap required, opened a new log
I20260926 22:50:27.528962 25129 raft_consensus.cc:348] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37aa0daf0f949c8baaf0cb1659f78d5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40591 } }
I20260926 22:50:27.529124 25129 raft_consensus.cc:374] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:50:27.529160 25129 raft_consensus.cc:729] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37aa0daf0f949c8baaf0cb1659f78d5, State: Initialized, Role: FOLLOWER
I20260926 22:50:27.529328 25129 consensus_queue.cc:260] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37aa0daf0f949c8baaf0cb1659f78d5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40591 } }
I20260926 22:50:27.529412 25129 raft_consensus.cc:388] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:50:27.529443 25129 raft_consensus.cc:482] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:50:27.529510 25129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:50:27.530160 25129 raft_consensus.cc:504] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37aa0daf0f949c8baaf0cb1659f78d5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40591 } }
I20260926 22:50:27.530277 25129 leader_election.cc:302] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c37aa0daf0f949c8baaf0cb1659f78d5; no voters:
I20260926 22:50:27.530431 25129 leader_election.cc:288] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 26 22:50:27 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:50:27.530488 25134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:50:27.530720 25134 raft_consensus.cc:686] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [term 1 LEADER]: Becoming Leader. State: Replica: c37aa0daf0f949c8baaf0cb1659f78d5, State: Running, Role: LEADER
I20260926 22:50:27.530798 25129 sys_catalog.cc:564] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:50:27.530851 25134 consensus_queue.cc:237] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37aa0daf0f949c8baaf0cb1659f78d5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40591 } }
I20260926 22:50:27.531169 25134 sys_catalog.cc:455] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c37aa0daf0f949c8baaf0cb1659f78d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37aa0daf0f949c8baaf0cb1659f78d5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40591 } } }
I20260926 22:50:27.531225 25134 sys_catalog.cc:458] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:50:27.531332 25134 sys_catalog.cc:455] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c37aa0daf0f949c8baaf0cb1659f78d5. Latest consensus state: current_term: 1 leader_uuid: "c37aa0daf0f949c8baaf0cb1659f78d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37aa0daf0f949c8baaf0cb1659f78d5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40591 } } }
I20260926 22:50:27.531364 25134 sys_catalog.cc:458] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:50:27.531597 25138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:50:27.532073 25138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:50:27.532357 25129 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 22:50:27.537274 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:27.523324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.13.148.62:40591)
0926 22:50:27.523610 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:27.523620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:27.523653 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:50:27.524198 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:27.524208 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:27.524274 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:27.524420 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:50:27.525164 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:27.526547 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:27.527566 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:27.528191 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:27.528293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:27.530993 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:50:27.530997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:50:27.530997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:50:27.531008 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:50:27.532861 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:27.534182 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:27.534238 (+ 56us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:27.534239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:27.534695 (+ 456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:27.534896 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:27.534897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:27.534897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:27.535593 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:50:27.535987 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:27.536461 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 22:50:27.536946 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":74,"threads_started":1}
I20260926 22:50:27.537313 25151 mvcc.cc:205] Tried to move back new op lower bound from 7333736560789401600 to 7333736560766820352. Current Snapshot: MvccSnapshot[applied={T|T < 7333736560789401600}]
I20260926 22:50:27.539863 25138 catalog_manager.cc:1347] Generated new cluster ID: 11c14faeadb24db7b5944ec32fe2ed51
I20260926 22:50:27.539911 25138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:50:27.549959 25138 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:50:27.550668 25138 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:50:27.558826 25138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5: Generated new TSK 0
I20260926 22:50:27.559046 25138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:50:28.256269 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:38277/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40591
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:50:28.372387 25174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:50:28.372675 25174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:50:28.372709 25174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:50:28.374434 25174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:50:28.374511 25174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:50:28.374550 25174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:50:28.376778 25174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.148.20:38277/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:50:28.377045 25174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:50:28.377341 25174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:50:28.382568 25180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:50:28.382575 25179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:50:28.382730 25174 server_base.cc:1034] running on GCE node
W20260926 22:50:28.382843 25182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:50:28.383028 25174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:50:28.383265 25174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:50:28.385038 25174 hybrid_clock.cc:648] HybridClock initialized: now 1790463028385022 us; error 30 us; skew 500 ppm
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:50:28.388139 25174 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:50:28.388639 25174 webserver.cc:460] Webserver started at http://127.13.148.1:43609/ using document root <none> and password file <none>
I20260926 22:50:28.388895 25174 fs_manager.cc:359] Metadata directory not provided
I20260926 22:50:28.388952 25174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:50:28.389076 25174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:50:28.403826 25174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "d3475246c94649ef9d3b98d68f9d3749"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "b4e311722e35bed662e424eb4f92bde3"
server_key_iv: "5a04814e8adfb2e11214cc0d74559e3a"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.404304 25174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "d3475246c94649ef9d3b98d68f9d3749"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "b4e311722e35bed662e424eb4f92bde3"
server_key_iv: "5a04814e8adfb2e11214cc0d74559e3a"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.406956 25174 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 22:50:28.421154 25189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.421377 25174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:50:28.421479 25174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "d3475246c94649ef9d3b98d68f9d3749"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "b4e311722e35bed662e424eb4f92bde3"
server_key_iv: "5a04814e8adfb2e11214cc0d74559e3a"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.421617 25174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:50:28.443838 25174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:50:28.446321 25174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:50:28.446540 25174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:50:28.446831 25174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:50:28.447221 25174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:50:28.447299 25174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.447391 25174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:50:28.447456 25174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.453322 25174 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:38827
I20260926 22:50:28.453831 25174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:50:28.458498 25304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:38827 every 8 connection(s)
I20260926 22:50:28.462198 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 25174
I20260926 22:50:28.462322 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:50:28.462534 13904 external_mini_cluster.cc:1428] Setting key 9ec93b58041f94fc48ce0ec165b897c9
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:50:28.470458 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:28.459191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40591 (local address 127.13.148.1:36939)
0926 22:50:28.459639 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:28.459685 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 22:50:28.460286 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:50:28.460696 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:50:28.460701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:50:28.460936 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:50:28.461221 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:28.461227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:28.462418 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:28.462419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:28.464299 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:28.464302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:28.464832 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:28.464833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:28.464968 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:28.465550 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:50:28.465561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:50:28.466964 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:50:28.468821 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:50:28.468828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:50:28.468830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:50:28.469109 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:50:28.469319 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:50:28.469320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:50:28.469320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:50:28.469452 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:50:28.469777 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:50:28.469784 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:50:28.469953 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 22:50:28.470060 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 22:50:28.470218 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":76,"threads_started":1}
I20260926 22:50:28.471083 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:28.459480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:36939 (local address 127.13.148.62:40591)
0926 22:50:28.460469 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:28.460471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:28.460478 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:50:28.460502 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:28.460503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:28.460515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:28.460572 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:50:28.461326 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:28.462208 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:28.464398 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:28.464714 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:28.464770 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:28.467059 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:50:28.467061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:50:28.467061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:50:28.467068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:50:28.468729 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:28.469189 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:28.469191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:28.469191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:28.469255 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:28.469534 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:28.469535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:28.469536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:28.469675 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:50:28.470852 (+ 1177us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:28.470897 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 22:50:28.470983 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":91,"threads_started":1}
I20260926 22:50:28.471639 25305 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40591
I20260926 22:50:28.471735 25305 heartbeater.cc:464] Registering TS with master...
I20260926 22:50:28.471902 25305 heartbeater.cc:511] Master 127.13.148.62:40591 requested a full tablet report, sending...
I20260926 22:50:28.472410 25093 ts_manager.cc:194] Registered new tserver with Master: d3475246c94649ef9d3b98d68f9d3749 (127.13.148.1:38827)
I20260926 22:50:28.472975 25093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:36939
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:50:28.494863 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:38277/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40591
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:50:28.600299 25311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:50:28.600601 25311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:50:28.600638 25311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:50:28.602324 25311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:50:28.602383 25311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:50:28.602421 25311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:50:28.604503 25311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.148.20:38277/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:50:28.604735 25311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:50:28.605065 25311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:50:28.610390 25318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:50:28.610405 25319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:50:28.610455 25321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:50:28.610738 25311 server_base.cc:1034] running on GCE node
I20260926 22:50:28.612882 25311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:50:28.613169 25311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:50:28.614517 25311 hybrid_clock.cc:648] HybridClock initialized: now 1790463028614492 us; error 38 us; skew 500 ppm
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:50:28.617478 25311 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:50:28.618008 25311 webserver.cc:460] Webserver started at http://127.13.148.2:40313/ using document root <none> and password file <none>
I20260926 22:50:28.618268 25311 fs_manager.cc:359] Metadata directory not provided
I20260926 22:50:28.618335 25311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:50:28.618508 25311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:50:28.634389 25311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "b5bc00915f75652cd75cecefc10cfb5b"
server_key_iv: "1dac9b920666b7f61d16985e67c86e80"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.634918 25311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "b5bc00915f75652cd75cecefc10cfb5b"
server_key_iv: "1dac9b920666b7f61d16985e67c86e80"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.638103 25311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 22:50:28.659353 25328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.662206 25311 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260926 22:50:28.662336 25311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "b5bc00915f75652cd75cecefc10cfb5b"
server_key_iv: "1dac9b920666b7f61d16985e67c86e80"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.662443 25311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:50:28.686615 25311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:50:28.688807 25311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:50:28.688962 25311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:50:28.689189 25311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:50:28.689574 25311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:50:28.689627 25311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.689657 25311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:50:28.689723 25311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.696112 25311 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:38889
I20260926 22:50:28.696625 25311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:50:28.698863 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 25311
I20260926 22:50:28.698979 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:50:28.699172 13904 external_mini_cluster.cc:1428] Setting key 9f962abb755f4f06fd76c6c5eb26d171
I20260926 22:50:28.702071 25441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:38889 every 8 connection(s)
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:50:28.715787 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:28.703014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:42453 (local address 127.13.148.62:40591)
0926 22:50:28.703059 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:28.703061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:28.706074 (+ 3013us) server_negotiation.cc:407] Connection header received
0926 22:50:28.706575 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:28.706576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:28.706589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:28.706637 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:50:28.707448 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:28.708211 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:28.709772 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:28.710079 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:28.710126 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:28.712622 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:50:28.712624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:50:28.712624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:50:28.712632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:50:28.714236 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:28.714723 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:28.714725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:28.714725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:28.714781 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:28.715046 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:28.715047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:28.715047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:28.715195 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:50:28.715237 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:28.715601 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 22:50:28.715690 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:50:28.716297 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:28.702737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40591 (local address 127.13.148.2:42453)
0926 22:50:28.705936 (+ 3199us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:28.705987 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 22:50:28.706486 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:50:28.706777 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:50:28.706781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:50:28.707025 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:50:28.707336 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:28.707345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:28.708327 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:28.708328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:28.709645 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:28.709648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:28.710254 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:28.710255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:28.710394 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:28.710973 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:50:28.710984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:50:28.712520 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:50:28.714328 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:50:28.714334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:50:28.714336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:50:28.714636 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:50:28.714845 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:50:28.714846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:50:28.714846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:50:28.714969 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:50:28.715284 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:50:28.715290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:50:28.715454 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 22:50:28.715880 (+ 426us) client_negotiation.cc:240] Negotiation successful
0926 22:50:28.716058 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3035,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260926 22:50:28.716974 25442 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40591
I20260926 22:50:28.717065 25442 heartbeater.cc:464] Registering TS with master...
I20260926 22:50:28.717258 25442 heartbeater.cc:511] Master 127.13.148.62:40591 requested a full tablet report, sending...
I20260926 22:50:28.717660 25093 ts_manager.cc:194] Registered new tserver with Master: 7fef7a1f1f7b404aa6f7a9b876ffb35d (127.13.148.2:38889)
I20260926 22:50:28.718169 25093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:42453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:50:28.748775 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:38277/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40591
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:50:28.869642 25448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:50:28.870041 25448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:50:28.870100 25448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:50:28.871850 25448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:50:28.871905 25448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:50:28.871942 25448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:50:28.874172 25448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.148.20:38277/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:50:28.874423 25448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:50:28.874771 25448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:50:28.881171 25453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:50:28.881178 25454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:50:28.881543 25456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:50:28.881899 25448 server_base.cc:1034] running on GCE node
I20260926 22:50:28.882089 25448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:50:28.882315 25448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:50:28.889633 25448 hybrid_clock.cc:648] HybridClock initialized: now 1790463028889608 us; error 35 us; skew 500 ppm
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:50:28.892809 25448 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:50:28.893369 25448 webserver.cc:460] Webserver started at http://127.13.148.3:46453/ using document root <none> and password file <none>
I20260926 22:50:28.893667 25448 fs_manager.cc:359] Metadata directory not provided
I20260926 22:50:28.893764 25448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:50:28.893909 25448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:50:28.910602 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "e446e655af1b4452ab1c04b4d400deb7"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "377058b9d973bfe53752850c72cd38b0"
server_key_iv: "0d6bdce632219cdb679e3a5d805b1c91"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.911068 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "e446e655af1b4452ab1c04b4d400deb7"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "377058b9d973bfe53752850c72cd38b0"
server_key_iv: "0d6bdce632219cdb679e3a5d805b1c91"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.918565 25448 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260926 22:50:28.936861 25463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.937108 25448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:50:28.938345 25448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "e446e655af1b4452ab1c04b4d400deb7"
format_stamp: "Formatted at 2026-09-26 22:50:28 on dist-test-slave-98hb"
server_key: "377058b9d973bfe53752850c72cd38b0"
server_key_iv: "0d6bdce632219cdb679e3a5d805b1c91"
server_key_version: "kuduclusterkey@0"
I20260926 22:50:28.938469 25448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:50:28.974255 25448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:50:28.977869 25448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:50:28.978015 25448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:50:28.978223 25448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:50:28.978562 25448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:50:28.978605 25448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.978632 25448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:50:28.978652 25448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:50:28.989306 25448 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:46427
I20260926 22:50:28.989877 25448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:50:28.996114 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 25448
I20260926 22:50:28.996233 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:50:28.996232 25576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:46427 every 8 connection(s)
I20260926 22:50:28.996446 13904 external_mini_cluster.cc:1428] Setting key 1d5a7293f35995cf1d78af2658e7129a
Sep 26 22:50:28 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:50:29.010679 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:28.997697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:38577 (local address 127.13.148.62:40591)
0926 22:50:28.997767 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:28.997768 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:28.998118 (+ 350us) server_negotiation.cc:407] Connection header received
0926 22:50:28.998769 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:28.998770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:28.998781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:28.998808 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:50:28.999617 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.000313 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.001865 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.002157 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.002195 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.005633 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:50:29.005635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:50:29.005635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:50:29.005643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:50:29.007318 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:29.007815 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:29.007816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:29.007817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:29.007879 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:29.008180 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:29.008181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:29.008181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:29.008315 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:50:29.008359 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:29.010482 (+ 2123us) server_negotiation.cc:299] Negotiation successful
0926 22:50:29.010580 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:50:29.011492 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:28.997723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40591 (local address 127.13.148.3:38577)
0926 22:50:28.998004 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:28.998045 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:50:28.998660 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:50:28.999001 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:50:28.999006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:50:28.999232 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:50:28.999516 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:28.999523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.000430 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.000431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.001778 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.001781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.002275 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.002276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.002410 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.003806 (+ 1396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:50:29.003816 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:50:29.005516 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:50:29.007414 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:50:29.007422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:50:29.007424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:50:29.007720 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:50:29.007952 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:50:29.007953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:50:29.007953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:50:29.008102 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:50:29.008541 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:50:29.008548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:50:29.008714 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 22:50:29.011062 (+ 2348us) client_negotiation.cc:240] Negotiation successful
0926 22:50:29.011241 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260926 22:50:29.012159 25577 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40591
I20260926 22:50:29.012256 25577 heartbeater.cc:464] Registering TS with master...
I20260926 22:50:29.012446 25577 heartbeater.cc:511] Master 127.13.148.62:40591 requested a full tablet report, sending...
I20260926 22:50:29.012857 25093 ts_manager.cc:194] Registered new tserver with Master: e446e655af1b4452ab1c04b4d400deb7 (127.13.148.3:46427)
I20260926 22:50:29.013350 25093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:38577
I20260926 22:50:29.018939 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:50:29.030229 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.020658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.13.148.62:40591)
0926 22:50:29.020699 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:29.020701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:29.021010 (+ 309us) server_negotiation.cc:407] Connection header received
0926 22:50:29.021110 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:29.021111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:29.021121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:29.021149 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:50:29.021957 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.022708 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.024051 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.024338 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.024384 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.027286 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:50:29.027287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:50:29.027288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:50:29.027295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:50:29.028894 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:29.029320 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:29.029321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:29.029322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:29.029379 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:50:29.029578 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:50:29.029578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:50:29.029579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:50:29.029701 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:50:29.029825 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:29.030057 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 22:50:29.030134 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:50:29.032513 25093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:50:29.034898 25093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:50:29.047410 25590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.038048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:38827 (local address 127.0.0.1:41138)
0926 22:50:29.039512 (+ 1464us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:29.039551 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:50:29.039608 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:50:29.040368 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:50:29.040369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:50:29.040370 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:50:29.040404 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.040405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.043541 (+ 3136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.043542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.045167 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.045170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.047094 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.047095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.047193 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.047207 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 22:50:29.047269 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:50:29.047321 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1424,"thread_start_us":83,"threads_started":1}
I20260926 22:50:29.048924 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.038922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.13.148.1:38827)
0926 22:50:29.039817 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:29.039822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:29.039831 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:50:29.039861 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:29.039864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:29.039879 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:29.039939 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:50:29.040461 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.043444 (+ 2983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.045813 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.046931 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.047001 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.047062 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:29.048795 (+ 1733us) server_negotiation.cc:299] Negotiation successful
0926 22:50:29.048841 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":68,"threads_started":1}
I20260926 22:50:29.049002 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.038252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46427 (local address 127.0.0.1:46462)
0926 22:50:29.039694 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:29.039701 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 22:50:29.039731 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:50:29.040245 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:50:29.040248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:50:29.040250 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:50:29.040311 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.040313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.046647 (+ 6334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.046648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.047741 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.047744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.048760 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.048761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.048829 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.048833 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:50:29.048887 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 22:50:29.048927 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"thread_start_us":49,"threads_started":1}
I20260926 22:50:29.049441 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.039072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.13.148.3:46427)
0926 22:50:29.040052 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:29.040056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:29.040064 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:50:29.040096 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:29.040097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:29.040110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:29.040168 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:50:29.041637 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.045705 (+ 4068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.047858 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.048620 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.048685 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.048752 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:29.049327 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 22:50:29.049366 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":75,"threads_started":1}
I20260926 22:50:29.050287 25239 tablet_service.cc:1467] Processing CreateTablet for tablet 8eea5c7d439f4da9914013df36088692 (DEFAULT_TABLE table=test-table [id=11ea85f17a6f4e748d12e61a72e58fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:50:29.050642 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eea5c7d439f4da9914013df36088692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:50:29.054796 25510 tablet_service.cc:1467] Processing CreateTablet for tablet 8eea5c7d439f4da9914013df36088692 (DEFAULT_TABLE table=test-table [id=11ea85f17a6f4e748d12e61a72e58fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:50:29.055116 25510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eea5c7d439f4da9914013df36088692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:50:29.057860 25595 tablet_bootstrap.cc:492] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749: Bootstrap starting.
I20260926 22:50:29.058539 25595 tablet_bootstrap.cc:654] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749: Neither blocks nor log segments found. Creating new log.
I20260926 22:50:29.058851 25595 log.cc:824] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749: Log is configured to *not* fsync() on all Append() calls
I20260926 22:50:29.063022 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.038392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:38889 (local address 127.0.0.1:41688)
0926 22:50:29.045286 (+ 6894us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:29.045297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:50:29.045334 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:50:29.059078 (+ 13744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:50:29.059079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:50:29.059083 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:50:29.059132 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.059134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.060671 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.060672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.061858 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.061861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.062793 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.062794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.062857 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.062864 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 22:50:29.062890 (+ 26us) client_negotiation.cc:240] Negotiation successful
0926 22:50:29.062928 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6877,"thread_start_us":57,"threads_started":1}
I20260926 22:50:29.063817 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.038523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.13.148.2:38889)
0926 22:50:29.058786 (+ 20263us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:29.058791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:29.058803 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:50:29.058841 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:29.058843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:29.058858 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:29.059206 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:50:29.059303 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.060577 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.061969 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.062700 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.063593 (+ 893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.063679 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:29.063703 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 22:50:29.063740 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20215,"thread_start_us":92,"threads_started":1}
I20260926 22:50:29.064100 25596 tablet_bootstrap.cc:492] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7: Bootstrap starting.
I20260926 22:50:29.064756 25372 tablet_service.cc:1467] Processing CreateTablet for tablet 8eea5c7d439f4da9914013df36088692 (DEFAULT_TABLE table=test-table [id=11ea85f17a6f4e748d12e61a72e58fa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:50:29.065099 25372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eea5c7d439f4da9914013df36088692. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:50:29.065342 25596 tablet_bootstrap.cc:654] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7: Neither blocks nor log segments found. Creating new log.
I20260926 22:50:29.065672 25596 log.cc:824] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:50:29.066481 25596 tablet_bootstrap.cc:492] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7: No bootstrap required, opened a new log
I20260926 22:50:29.066550 25596 ts_tablet_manager.cc:1397] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:50:29.068423 25596 raft_consensus.cc:348] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.068580 25596 raft_consensus.cc:374] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:50:29.068608 25596 raft_consensus.cc:729] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e446e655af1b4452ab1c04b4d400deb7, State: Initialized, Role: FOLLOWER
I20260926 22:50:29.068712 25596 consensus_queue.cc:260] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.068938 25596 ts_tablet_manager.cc:1428] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:50:29.069252 25577 heartbeater.cc:503] Master 127.13.148.62:40591 was elected leader, sending a full tablet report...
I20260926 22:50:29.069587 25595 tablet_bootstrap.cc:492] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749: No bootstrap required, opened a new log
I20260926 22:50:29.069644 25595 ts_tablet_manager.cc:1397] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260926 22:50:29.072104 25595 raft_consensus.cc:348] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.072268 25595 raft_consensus.cc:374] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:50:29.072304 25595 raft_consensus.cc:729] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3475246c94649ef9d3b98d68f9d3749, State: Initialized, Role: FOLLOWER
I20260926 22:50:29.072434 25595 consensus_queue.cc:260] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.072700 25595 ts_tablet_manager.cc:1428] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:50:29.072808 25305 heartbeater.cc:503] Master 127.13.148.62:40591 was elected leader, sending a full tablet report...
I20260926 22:50:29.077108 25599 tablet_bootstrap.cc:492] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Bootstrap starting.
I20260926 22:50:29.078099 25599 tablet_bootstrap.cc:654] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Neither blocks nor log segments found. Creating new log.
I20260926 22:50:29.078629 25599 log.cc:824] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:50:29.079838 25599 tablet_bootstrap.cc:492] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: No bootstrap required, opened a new log
I20260926 22:50:29.079907 25599 ts_tablet_manager.cc:1397] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:50:29.081800 25599 raft_consensus.cc:348] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.081965 25599 raft_consensus.cc:374] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:50:29.082005 25599 raft_consensus.cc:729] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fef7a1f1f7b404aa6f7a9b876ffb35d, State: Initialized, Role: FOLLOWER
I20260926 22:50:29.082109 25599 consensus_queue.cc:260] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.082353 25599 ts_tablet_manager.cc:1428] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:50:29.082504 25442 heartbeater.cc:503] Master 127.13.148.62:40591 was elected leader, sending a full tablet report...
I20260926 22:50:29.132858 25601 raft_consensus.cc:482] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:50:29.133021 25601 raft_consensus.cc:504] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.133430 25601 leader_election.cc:288] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e446e655af1b4452ab1c04b4d400deb7 (127.13.148.3:46427), 7fef7a1f1f7b404aa6f7a9b876ffb35d (127.13.148.2:38889)
Sep 26 22:50:29 dist-test-slave-98hb krb5kdc[24617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463028, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:50:29.148134 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.134003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:54863 (local address 127.13.148.2:38889)
0926 22:50:29.134163 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:29.134165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:29.141922 (+ 7757us) server_negotiation.cc:407] Connection header received
0926 22:50:29.142031 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:29.142032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:29.142046 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:29.142075 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:50:29.142361 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.143124 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.147180 (+ 4056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.147752 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.147801 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.147810 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:29.148015 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 22:50:29.148055 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 22:50:29.148393 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.133638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:38889 (local address 127.13.148.1:54863)
0926 22:50:29.141840 (+ 8202us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:29.141854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:50:29.141955 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:50:29.142187 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:50:29.142189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:50:29.142202 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:50:29.142287 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.142289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.143284 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.143285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.144575 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.144579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.147866 (+ 3287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.147867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.147941 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.147944 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:50:29.148264 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 22:50:29.148309 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8177,"thread_start_us":814,"threads_started":1}
I20260926 22:50:29.148758 25396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eea5c7d439f4da9914013df36088692" candidate_uuid: "d3475246c94649ef9d3b98d68f9d3749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" is_pre_election: true
I20260926 22:50:29.148924 25396 raft_consensus.cc:2454] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3475246c94649ef9d3b98d68f9d3749 in term 0.
I20260926 22:50:29.149160 25191 leader_election.cc:302] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fef7a1f1f7b404aa6f7a9b876ffb35d, d3475246c94649ef9d3b98d68f9d3749; no voters:
I20260926 22:50:29.149305 25601 raft_consensus.cc:2781] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:50:29.149353 25601 raft_consensus.cc:482] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:50:29.149379 25601 raft_consensus.cc:3037] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:50:29.150327 25601 raft_consensus.cc:504] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.150470 25601 leader_election.cc:288] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [CANDIDATE]: Term 1 election: Requested vote from peers e446e655af1b4452ab1c04b4d400deb7 (127.13.148.3:46427), 7fef7a1f1f7b404aa6f7a9b876ffb35d (127.13.148.2:38889)
I20260926 22:50:29.150686 25396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eea5c7d439f4da9914013df36088692" candidate_uuid: "d3475246c94649ef9d3b98d68f9d3749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d"
I20260926 22:50:29.150764 25396 raft_consensus.cc:3037] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:50:29.151598 25396 raft_consensus.cc:2454] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3475246c94649ef9d3b98d68f9d3749 in term 1.
I20260926 22:50:29.152022 25191 leader_election.cc:302] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fef7a1f1f7b404aa6f7a9b876ffb35d, d3475246c94649ef9d3b98d68f9d3749; no voters:
I20260926 22:50:29.152127 25601 raft_consensus.cc:2781] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:50:29.152271 25601 raft_consensus.cc:686] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [term 1 LEADER]: Becoming Leader. State: Replica: d3475246c94649ef9d3b98d68f9d3749, State: Running, Role: LEADER
I20260926 22:50:29.152371 25601 consensus_queue.cc:237] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } }
I20260926 22:50:29.153510 25093 catalog_manager.cc:5495] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3475246c94649ef9d3b98d68f9d3749 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "d3475246c94649ef9d3b98d68f9d3749" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3475246c94649ef9d3b98d68f9d3749" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 38827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 } health_report { overall_health: UNKNOWN } } }
I20260926 22:50:29.167516 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.134055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:36443 (local address 127.13.148.3:46427)
0926 22:50:29.134196 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:29.134199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:29.144753 (+ 10554us) server_negotiation.cc:407] Connection header received
0926 22:50:29.144847 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:29.144848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:29.144858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:29.144889 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:50:29.145080 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.145835 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.160805 (+ 14970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.161415 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.161514 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.161525 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:29.167347 (+ 5822us) server_negotiation.cc:299] Negotiation successful
0926 22:50:29.167431 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 22:50:29.167526 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.133642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46427 (local address 127.13.148.1:36443)
0926 22:50:29.144698 (+ 11056us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:29.144708 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:50:29.144779 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:50:29.144946 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:50:29.144947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:50:29.144949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:50:29.145012 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.145013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.145950 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.145951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.147073 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:50:29.147076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.162776 (+ 15700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.162778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:50:29.162869 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.162875 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 22:50:29.167327 (+ 4452us) client_negotiation.cc:240] Negotiation successful
0926 22:50:29.167391 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11038,"thread_start_us":135,"threads_started":1}
I20260926 22:50:29.168393 25515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eea5c7d439f4da9914013df36088692" candidate_uuid: "d3475246c94649ef9d3b98d68f9d3749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e446e655af1b4452ab1c04b4d400deb7" is_pre_election: true
I20260926 22:50:29.168577 25515 raft_consensus.cc:2454] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3475246c94649ef9d3b98d68f9d3749 in term 0.
I20260926 22:50:29.168690 25528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eea5c7d439f4da9914013df36088692" candidate_uuid: "d3475246c94649ef9d3b98d68f9d3749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e446e655af1b4452ab1c04b4d400deb7"
I20260926 22:50:29.168747 25528 raft_consensus.cc:3037] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:50:29.169930 25528 raft_consensus.cc:2454] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3475246c94649ef9d3b98d68f9d3749 in term 1.
W20260926 22:50:29.203514 25443 tablet.cc:2369] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:50:29.203833 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.199943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.13.148.1:38827)
0926 22:50:29.200089 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 22:50:29.200092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:50:29.200099 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:50:29.200137 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:50:29.200138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:50:29.200160 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:50:29.200232 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:50:29.200404 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.201173 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.202708 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:50:29.203011 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:50:29.203074 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:50:29.203266 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:50:29.203399 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:50:29.203477 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:50:29.203715 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 22:50:29.203738 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260926 22:50:29.205652 25396 raft_consensus.cc:1261] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 1 FOLLOWER]: Refusing update from remote peer d3475246c94649ef9d3b98d68f9d3749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:50:29.205938 25528 raft_consensus.cc:1261] T 8eea5c7d439f4da9914013df36088692 P e446e655af1b4452ab1c04b4d400deb7 [term 1 FOLLOWER]: Refusing update from remote peer d3475246c94649ef9d3b98d68f9d3749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:50:29.206286 25606 consensus_queue.cc:1035] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e446e655af1b4452ab1c04b4d400deb7" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:50:29.206806 25606 consensus_queue.cc:1035] T 8eea5c7d439f4da9914013df36088692 P d3475246c94649ef9d3b98d68f9d3749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fef7a1f1f7b404aa6f7a9b876ffb35d" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:50:29.216677 25609 mvcc.cc:205] Tried to move back new op lower bound from 7333736567623368704 to 7333736567408766976. Current Snapshot: MvccSnapshot[applied={T|T < 7333736567623368704}]
I20260926 22:50:29.216805 25610 mvcc.cc:205] Tried to move back new op lower bound from 7333736567623368704 to 7333736567408766976. Current Snapshot: MvccSnapshot[applied={T|T < 7333736567623368704}]
I20260926 22:50:29.217708 25612 mvcc.cc:205] Tried to move back new op lower bound from 7333736567623368704 to 7333736567408766976. Current Snapshot: MvccSnapshot[applied={T|T < 7333736567623368704}]
I20260926 22:50:29.220389 25093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:50:29.220491 25093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:50:29.223651 25093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c37aa0daf0f949c8baaf0cb1659f78d5: Sending DeleteTablet for 3 replicas of tablet 8eea5c7d439f4da9914013df36088692
I20260926 22:50:29.224201 25372 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eea5c7d439f4da9914013df36088692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:50:29 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:41688
I20260926 22:50:29.224846 25510 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eea5c7d439f4da9914013df36088692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:50:29 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46462
I20260926 22:50:29.228820 25239 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eea5c7d439f4da9914013df36088692 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:50:29 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:41138
I20260926 22:50:29.229579 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 25174
I20260926 22:50:29.230748 25620 tablet_replica.cc:330] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: stopping tablet replica
I20260926 22:50:29.231169 25620 raft_consensus.cc:2229] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:50:29.231287 25620 raft_consensus.cc:2258] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:50:29.231851 25620 ts_tablet_manager.cc:1905] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:50:29.233688 25620 ts_tablet_manager.cc:1918] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:50:29.233768 25620 log.cc:1196] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/8eea5c7d439f4da9914013df36088692
I20260926 22:50:29.234000 25620 ts_tablet_manager.cc:1939] T 8eea5c7d439f4da9914013df36088692 P 7fef7a1f1f7b404aa6f7a9b876ffb35d: Deleting consensus metadata
I20260926 22:50:29.234387 25078 catalog_manager.cc:4841] TS 7fef7a1f1f7b404aa6f7a9b876ffb35d (127.13.148.2:38889): tablet 8eea5c7d439f4da9914013df36088692 (table test-table [id=11ea85f17a6f4e748d12e61a72e58fa6]) successfully deleted
W20260926 22:50:29.236975 25080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:38827 (error 108)
I20260926 22:50:29.237556 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 25311
I20260926 22:50:29.238577 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.238468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:38827 (local address 127.0.0.1:41166)
0926 22:50:29.238514 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:29.238543 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:38827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 22:50:29.238768 25080 catalog_manager.cc:4584] TS d3475246c94649ef9d3b98d68f9d3749 (127.13.148.1:38827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8eea5c7d439f4da9914013df36088692: Network error: Client connection negotiation failed: client connection to 127.13.148.1:38827: connect: Connection refused (error 111)
I20260926 22:50:29.242863 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 25448
I20260926 22:50:29.248006 25589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:50:29.247889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46427 (local address 127.0.0.1:46472)
0926 22:50:29.247933 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:50:29.247955 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.3:46427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 22:50:29.247995 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 25061
I20260926 22:50:29.253786 13904 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 22:50:29.587076 13904 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 22:50:29.587170 13904 mini_ranger.cc:66] Stopping Ranger...
I20260926 22:50:29.920342 13904 mini_ranger.cc:68] Stopped Ranger
2026-09-26 22:50:29.920 UTC [24647] LOG: received smart shutdown request
2026-09-26 22:50:29.924 UTC [24647] LOG: background worker "logical replication launcher" (PID 24656) exited with exit code 1
2026-09-26 22:50:29.924 UTC [24651] LOG: shutting down
2026-09-26 22:50:30.070 UTC [24647] LOG: database system is shut down
2026-09-26T22:50:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43165 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:50:30 dist-test-slave-98hb krb5kdc[25630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:50:30 dist-test-slave-98hb krb5kdc[25630](info): set up 2 sockets
Sep 26 22:50:30 dist-test-slave-98hb krb5kdc[25630](info): commencing operation
krb5kdc: starting...
W20260926 22:50:31.097698 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:50:31 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:50:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/rangeradmin_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/rangeradmin_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/rangerlookup_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/rangerlookup_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
I20260926 22:50:31.256048 13904 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskClAdwy/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/postgres -l logfile start
2026-09-26 22:50:33.528 UTC [25660] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 22:50:33.528 UTC [25660] LOG: listening on IPv4 address "127.13.148.20", port 51765
2026-09-26 22:50:33.531 UTC [25660] LOG: listening on Unix socket "/tmp/.s.PGSQL.51765"
2026-09-26 22:50:33.558 UTC [25663] LOG: database system was shut down at 2026-09-26 22:50:32 UTC
2026-09-26 22:50:33.561 UTC [25660] LOG: database system is ready to accept connections
I20260926 22:50:34.421686 13904 mini_postgres.cc:96] Postgres bound to 51765
2026-09-26 22:50:34.427 UTC [25671] FATAL: database "slave" does not exist
127.13.148.20:51765 - accepting connections
I20260926 22:50:34.427760 13904 mini_ranger.cc:161] Starting Ranger...
I20260926 22:50:34.441398 13904 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 22:50:34.855142 13904 mini_ranger.cc:87] Created ranger Postgres database
I20260926 22:50:34.855211 13904 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-admin
2026-09-26 22:50:35,119 [I] DB FLAVOR :POSTGRES
2026-09-26 22:50:35,119 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 22:50:35,119 [I] Checking connection..
2026-09-26 22:50:35,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 22:50:35,363 [I] Checking connection passed.
2026-09-26 22:50:35,363 [I] --------- Verifying version history table ---------
2026-09-26 22:50:35,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:50:35,606 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 22:50:35,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:50:35,857 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 22:50:35,857 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 22:50:35,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 22:50:36.085 UTC [25735] WARNING: there is no transaction in progress
2026-09-26 22:50:36,090 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 22:50:36,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 22:50:36,328 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 22:50:36,328 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 22:50:36,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 22:50:36,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 22:50:36,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-98hb.c.gcp-upstream.internal','N') ;"
2026-09-26 22:50:37,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 22:50:37,300 [I] Table x_portal_user does not exist in database ranger
2026-09-26 22:50:37,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 22:50:37,549 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 22:50:37,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 22:50:37,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 22:50:38,001 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 22:50:38,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 22:50:38.859 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:38.874 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:38.888 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:38.901 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:38.916 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:38.935 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:38.986 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:38.992 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39.005 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39.014 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39.024 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39.037 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39.047 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39.053 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39.059 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39.064 UTC [25843] WARNING: there is no transaction in progress
2026-09-26 22:50:39,634 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 22:50:39,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 22:50:39,877 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 22:50:39,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 22:50:40,112 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 22:50:40,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 22:50:40,364 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 22:50:40,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 22:50:40,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 22:50:40,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-98hb.c.gcp-upstream.internal';"
2026-09-26 22:50:41,064 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 22:50:41,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 22:50:41,296 [I] DB_PATCHES have already been applied
I20260926 22:50:41.301666 13904 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-admin:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskClAdwy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 22:50:41.301726 13904 mini_ranger.cc:193] Using host: 127.13.148.20
I20260926 22:50:41.304579 13904 mini_ranger.cc:227] Ranger admin URL: http://127.13.148.20:45795
Sep 26, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 10:50:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 10:50:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 22:50:42.299727 13904 mini_ranger.cc:160] Time spent starting Ranger: real 7.872s user 0.004s sys 0.004s
Sep 26, 2026 10:51:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 10:51:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 10:51:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 10:51:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 10:51:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 22:51:03.852020 13904 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.148.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.148.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/rangerkms_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.148.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/rangerkms_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.148.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.148.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.148.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/HTTP_127.13.148.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 22:51:03 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463063, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 22:51:03.927963 13904 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 22:51:03.936031 13904 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 22:51:04.700425 13904 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 22:51:04.700496 13904 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms
I20260926 22:51:04.700513 13904 mini_ranger_kms.cc:227] Using postgres at 127.13.148.20:51765
2026-09-26 22:51:04,846 [I] DB FLAVOR :POSTGRES
2026-09-26 22:51:04,846 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 22:51:04,846 [I] Checking connection
2026-09-26 22:51:04,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 22:51:05,070 [I] connection success
2026-09-26 22:51:05,070 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 22:51:05,070 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 22:51:05,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 22:51:05,302 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 22:51:05,302 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 22:51:05,302 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 22:51:05,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.148.20:51765/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 22:51:05,568 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 22:51:06.030730 13904 mini_ranger_kms.cc:310] Created kms service
I20260926 22:51:06.179124 13904 mini_ranger_kms.cc:326] Created kudu user
I20260926 22:51:06.229931 13904 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 22:51:06 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463063, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:06.640043 13904 mini_ranger_kms.cc:384] Added ranger policy
I20260926 22:51:06.640256 13904 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms:/tmp/dist-test-taskClAdwy/build/release/bin/postgresql.jar:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskClAdwy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 22:51:06.640281 13904 mini_ranger_kms.cc:240] Using host: 127.13.148.20
I20260926 22:51:06.644014 13904 mini_ranger_kms.cc:276] Ranger KMS PID: 26023
I20260926 22:51:06.644074 13904 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.148.20:35449
Sep 26, 2026 10:51:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 10:51:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 10:51:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 10:51:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 10:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 10:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 22:51:08.630102 13904 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.702s user 0.007s sys 0.120s
I20260926 22:51:08.630198 13904 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 22:51:08.630242 13904 mini_ranger_kms.cc:401] 127.13.148.20:35449/kms/v1/keys
Sep 26 22:51:08 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790463068, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 22:51:09 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463068, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Sep 26 22:51:09 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463068, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.148.20@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
Sep 26 22:51:11 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:51:11.698371 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35853
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35853
--ranger_config_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:35449/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:11.776782 26074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:11.776978 26074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:11.776998 26074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:11.778177 26074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:11.778216 26074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:11.778229 26074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:11.778241 26074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:11.778264 26074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:11.779767 26074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.148.20:35449/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35853
--ranger_config_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:11.779980 26074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:11.780180 26074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:11.784941 26082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:11.784978 26080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:11.784991 26074 server_base.cc:1034] running on GCE node
W20260926 22:51:11.785084 26079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:11.785579 26074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:11.785898 26074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:11.787055 26074 hybrid_clock.cc:648] HybridClock initialized: now 1790463071787031 us; error 42 us; skew 500 ppm
Sep 26 22:51:11 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:11.793938 26074 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:51:11.794406 26074 webserver.cc:460] Webserver started at http://127.13.148.62:33267/ using document root <none> and password file <none>
I20260926 22:51:11.794611 26074 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:11.794649 26074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:11.794775 26074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:51:11 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463071, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:11.840410 26074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "257ac573577d40d48832c88eae17236f"
format_stamp: "Formatted at 2026-09-26 22:51:11 on dist-test-slave-98hb"
server_key: "9474384dad981825483281259cf7fb89"
server_key_iv: "52a1df7d0f05a82baac2c3adc0df493d"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:11.840796 26074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "257ac573577d40d48832c88eae17236f"
format_stamp: "Formatted at 2026-09-26 22:51:11 on dist-test-slave-98hb"
server_key: "9474384dad981825483281259cf7fb89"
server_key_iv: "52a1df7d0f05a82baac2c3adc0df493d"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:11.842147 26074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:11.856460 26089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:11.856714 26074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:11.856809 26074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "257ac573577d40d48832c88eae17236f"
format_stamp: "Formatted at 2026-09-26 22:51:11 on dist-test-slave-98hb"
server_key: "9474384dad981825483281259cf7fb89"
server_key_iv: "52a1df7d0f05a82baac2c3adc0df493d"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:11.856892 26074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:11.888748 26074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:11.890556 26074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:11.890700 26074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:11.894932 26074 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35853
I20260926 22:51:11.895002 26141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35853 every 8 connection(s)
I20260926 22:51:11.895328 26074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:51:11.895979 26142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:11.898216 26142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Bootstrap starting.
I20260926 22:51:11.898752 26142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:11.898999 26142 log.cc:824] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:11.899583 26142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: No bootstrap required, opened a new log
I20260926 22:51:11.901449 26142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } }
I20260926 22:51:11.901587 26142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:11.901618 26142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257ac573577d40d48832c88eae17236f, State: Initialized, Role: FOLLOWER
I20260926 22:51:11.901778 26142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } }
I20260926 22:51:11.901856 26142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:51:11.901887 26142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:51:11.901932 26142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:11.902413 26142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } }
I20260926 22:51:11.902503 26142 leader_election.cc:302] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 257ac573577d40d48832c88eae17236f; no voters:
I20260926 22:51:11.902650 26142 leader_election.cc:288] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:51:11.902683 26145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:11.902786 26145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 1 LEADER]: Becoming Leader. State: Replica: 257ac573577d40d48832c88eae17236f, State: Running, Role: LEADER
I20260926 22:51:11.902896 26142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:11.902922 26145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } }
I20260926 22:51:11.903537 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 26074
I20260926 22:51:11.903631 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:51:11.903789 13904 external_mini_cluster.cc:1428] Setting key be5e126787b2320f6218ab0fb6ddd1a3
I20260926 22:51:11.903854 26147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 257ac573577d40d48832c88eae17236f. Latest consensus state: current_term: 1 leader_uuid: "257ac573577d40d48832c88eae17236f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } } }
I20260926 22:51:11.903929 26147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:11.904176 26159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:51:11.904268 26142 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 22:51:11.904640 26159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:51:11.904819 26146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "257ac573577d40d48832c88eae17236f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } } }
I20260926 22:51:11.904949 26146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:11.913090 26159 catalog_manager.cc:1347] Generated new cluster ID: 048d60a4cff14cc58f13ad29a350f0ec
I20260926 22:51:11.913205 26159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:51:11 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:11.930347 26159 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:51:11.931114 26159 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:51:11.935142 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:11.904345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.13.148.62:35853)
0926 22:51:11.905269 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:11.905279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:11.905310 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:51:11.905425 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:11.905432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:11.905487 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:11.905615 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:11.906489 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:11.907896 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:11.908933 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:11.912006 (+ 3073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:11.919073 (+ 7067us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:11.921822 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:11.921827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:11.921827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:11.921840 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:11.923754 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:11.933813 (+ 10059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:11.933818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:11.933818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:11.933947 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:11.934186 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:11.934187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:11.934187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:11.934494 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:11.934665 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:11.934751 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 22:51:11.934906 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":74,"threads_started":1}
I20260926 22:51:11.938721 26159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Generated new TSK 0
I20260926 22:51:11.938937 26159 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:51:12.666874 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:35449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35853
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:12.783648 26187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:12.783918 26187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:12.783949 26187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:12.785609 26187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:12.785665 26187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:12.785701 26187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:51:12.787738 26187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.148.20:35449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:12.788215 26187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:12.788535 26187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:12.794543 26193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:12.794528 26192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:12.794788 26195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:12.795195 26187 server_base.cc:1034] running on GCE node
I20260926 22:51:12.795389 26187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:12.795634 26187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:12.796777 26187 hybrid_clock.cc:648] HybridClock initialized: now 1790463072796762 us; error 27 us; skew 500 ppm
Sep 26 22:51:12 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463072, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:12.799798 26187 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:51:12.800311 26187 webserver.cc:460] Webserver started at http://127.13.148.1:41415/ using document root <none> and password file <none>
I20260926 22:51:12.800590 26187 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:12.800662 26187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:12.800809 26187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:51:12 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463072, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:12.819279 26187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "de23bcce36fa4bc7a022dfd68112762f"
format_stamp: "Formatted at 2026-09-26 22:51:12 on dist-test-slave-98hb"
server_key: "adb9b3c24253d8479aca9019d9747e63"
server_key_iv: "5e6eeb37e31e2de3a780ed2c17b9df0e"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:12.819823 26187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "de23bcce36fa4bc7a022dfd68112762f"
format_stamp: "Formatted at 2026-09-26 22:51:12 on dist-test-slave-98hb"
server_key: "adb9b3c24253d8479aca9019d9747e63"
server_key_iv: "5e6eeb37e31e2de3a780ed2c17b9df0e"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:12.821614 26187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:12.834586 26202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:12.834801 26187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:51:12.834890 26187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "de23bcce36fa4bc7a022dfd68112762f"
format_stamp: "Formatted at 2026-09-26 22:51:12 on dist-test-slave-98hb"
server_key: "adb9b3c24253d8479aca9019d9747e63"
server_key_iv: "5e6eeb37e31e2de3a780ed2c17b9df0e"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:12.834985 26187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:12.853160 26187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:12.855590 26187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:12.855777 26187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:12.856060 26187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:12.856427 26187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:12.856506 26187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:12.856559 26187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:12.856593 26187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:12.864141 26187 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:46473
I20260926 22:51:12.864229 26317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:46473 every 8 connection(s)
I20260926 22:51:12.864619 26187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:51:12.866488 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 26187
I20260926 22:51:12.866585 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:51:12.866771 13904 external_mini_cluster.cc:1428] Setting key 879399e86879f26db0e0ba33f35e5449
Sep 26 22:51:12 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463072, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:12.880463 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:12.865422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35853 (local address 127.13.148.1:54761)
0926 22:51:12.870391 (+ 4969us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:12.870442 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:12.871052 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:12.871470 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:12.871474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:12.871717 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:12.871990 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:12.871996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:12.872957 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:12.872958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:12.874133 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:12.874136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:12.874603 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:12.874604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:12.874752 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:12.875291 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:12.875301 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:12.876772 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:12.878703 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:12.878711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:12.878713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:12.879018 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:12.879247 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:12.879248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:12.879249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:12.879399 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:12.879731 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:12.879738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:12.879916 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 22:51:12.880038 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 22:51:12.880203 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4921,"thread_start_us":96,"threads_started":1}
I20260926 22:51:12.880841 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:12.865675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:54761 (local address 127.13.148.62:35853)
0926 22:51:12.871240 (+ 5565us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:12.871242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:12.871250 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:51:12.871273 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:12.871273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:12.871285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:12.871343 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:12.872093 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:12.872847 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:12.874224 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:12.874476 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:12.874530 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:12.876868 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:12.876870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:12.876870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:12.876877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:12.878594 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:12.879105 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:12.879107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:12.879107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:12.879177 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:12.879477 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:12.879478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:12.879479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:12.879623 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:12.879702 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:12.880632 (+ 930us) server_negotiation.cc:299] Negotiation successful
0926 22:51:12.880739 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5535,"thread_start_us":100,"threads_started":1}
I20260926 22:51:12.881392 26318 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35853
I20260926 22:51:12.881500 26318 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:12.881682 26318 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:12.882272 26106 ts_manager.cc:194] Registered new tserver with Master: de23bcce36fa4bc7a022dfd68112762f (127.13.148.1:46473)
I20260926 22:51:12.883198 26106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:54761
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:51:12.908428 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:35449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35853
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:13.022370 26324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:13.022751 26324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:13.022850 26324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:13.024606 26324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:13.024715 26324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:13.024801 26324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:51:13.026994 26324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.148.20:35449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:13.027454 26324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:13.027806 26324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:13.034364 26332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:13.034718 26331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:13.034917 26334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:13.035691 26324 server_base.cc:1034] running on GCE node
I20260926 22:51:13.036024 26324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:13.036296 26324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:13.037465 26324 hybrid_clock.cc:648] HybridClock initialized: now 1790463073037441 us; error 38 us; skew 500 ppm
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:13.044268 26324 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:51:13.044879 26324 webserver.cc:460] Webserver started at http://127.13.148.2:45597/ using document root <none> and password file <none>
I20260926 22:51:13.045181 26324 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:13.045243 26324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:13.045370 26324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:13.065136 26324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "4601464d35194aa5b2f8cc7bb7dfacfb"
format_stamp: "Formatted at 2026-09-26 22:51:13 on dist-test-slave-98hb"
server_key: "905d8077c2091b40650ab7b731f47a28"
server_key_iv: "1a145672e0e9bbcb7725965bae8d4221"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.065603 26324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "4601464d35194aa5b2f8cc7bb7dfacfb"
format_stamp: "Formatted at 2026-09-26 22:51:13 on dist-test-slave-98hb"
server_key: "905d8077c2091b40650ab7b731f47a28"
server_key_iv: "1a145672e0e9bbcb7725965bae8d4221"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.068683 26324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:13.093900 26341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.094187 26324 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260926 22:51:13.094280 26324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "4601464d35194aa5b2f8cc7bb7dfacfb"
format_stamp: "Formatted at 2026-09-26 22:51:13 on dist-test-slave-98hb"
server_key: "905d8077c2091b40650ab7b731f47a28"
server_key_iv: "1a145672e0e9bbcb7725965bae8d4221"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.094377 26324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:13.144737 26324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:13.147131 26324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:13.147330 26324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:13.147622 26324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:13.148047 26324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:13.148128 26324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.148197 26324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:13.148250 26324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.155435 26324 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:46609
I20260926 22:51:13.155905 26324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:51:13.157785 26454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:46609 every 8 connection(s)
I20260926 22:51:13.159345 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 26324
I20260926 22:51:13.159444 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:51:13.159626 13904 external_mini_cluster.cc:1428] Setting key ba77aa5de823316a4f209d9d1bde5002
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:13.169869 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:13.158749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:51117 (local address 127.13.148.62:35853)
0926 22:51:13.158821 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:13.158824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:13.159061 (+ 237us) server_negotiation.cc:407] Connection header received
0926 22:51:13.159781 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:13.159782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:13.159798 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:13.159875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:13.161241 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.162075 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.163518 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.163844 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.163888 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:13.166542 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:13.166543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:13.166544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:13.166553 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:13.168188 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:13.168697 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:13.168699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:13.168699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:13.168770 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:13.169075 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:13.169076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:13.169076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:13.169217 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:13.169277 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:13.169618 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 22:51:13.169715 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:51:13.170044 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:13.158475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35853 (local address 127.13.148.2:51117)
0926 22:51:13.158913 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:13.158969 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:13.159662 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:13.160563 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:13.160568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:13.160798 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:13.161143 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:13.161149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.162183 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.162184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:13.163418 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:13.163420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.164007 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.164008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:13.164150 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:13.164729 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:13.164745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:13.166435 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:13.168274 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:13.168283 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:13.168286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:13.168606 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:13.168840 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:13.168842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:13.168842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:13.169009 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:13.169291 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:13.169298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:13.169480 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 22:51:13.169597 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 22:51:13.169794 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":82,"threads_started":1}
I20260926 22:51:13.170852 26455 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35853
I20260926 22:51:13.170950 26455 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:13.171154 26455 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:13.171537 26106 ts_manager.cc:194] Registered new tserver with Master: 4601464d35194aa5b2f8cc7bb7dfacfb (127.13.148.2:46609)
I20260926 22:51:13.172035 26106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:51117
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:51:13.214828 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:35449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35853
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:13.302085 26461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:13.302278 26461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:13.302307 26461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:13.303462 26461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:13.303505 26461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:13.303539 26461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:51:13.305037 26461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.148.20:35449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:13.305253 26461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:13.305480 26461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:13.310305 26469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:13.310463 26467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:13.310681 26461 server_base.cc:1034] running on GCE node
W20260926 22:51:13.310735 26466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:13.310951 26461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:13.311188 26461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:13.312340 26461 hybrid_clock.cc:648] HybridClock initialized: now 1790463073312321 us; error 35 us; skew 500 ppm
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:13.314739 26461 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:51:13.315116 26461 webserver.cc:460] Webserver started at http://127.13.148.3:35477/ using document root <none> and password file <none>
I20260926 22:51:13.315330 26461 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:13.315377 26461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:13.315507 26461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:13.331969 26461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "196a3d2593234026bfb72720852dab8c"
format_stamp: "Formatted at 2026-09-26 22:51:13 on dist-test-slave-98hb"
server_key: "707dbcf3bd48f791249a0007a68ae1b0"
server_key_iv: "c060ed8b000142bef684a6b0cf18db5d"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.332510 26461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "196a3d2593234026bfb72720852dab8c"
format_stamp: "Formatted at 2026-09-26 22:51:13 on dist-test-slave-98hb"
server_key: "707dbcf3bd48f791249a0007a68ae1b0"
server_key_iv: "c060ed8b000142bef684a6b0cf18db5d"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.338943 26461 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260926 22:51:13.358938 26476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.359200 26461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:13.359290 26461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "196a3d2593234026bfb72720852dab8c"
format_stamp: "Formatted at 2026-09-26 22:51:13 on dist-test-slave-98hb"
server_key: "707dbcf3bd48f791249a0007a68ae1b0"
server_key_iv: "c060ed8b000142bef684a6b0cf18db5d"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.359406 26461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:13.381029 26461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:13.382895 26461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:13.383033 26461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:13.383249 26461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:13.383574 26461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:13.383610 26461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.383639 26461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:13.383663 26461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.390195 26461 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:37341
I20260926 22:51:13.390285 26589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:37341 every 8 connection(s)
I20260926 22:51:13.390568 26461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:51:13.393046 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 26461
I20260926 22:51:13.393155 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:51:13.393326 13904 external_mini_cluster.cc:1428] Setting key 5a5796d99762ddbb0eb02a2d8ca0cb9a
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:13.409785 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:13.391321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:51589 (local address 127.13.148.62:35853)
0926 22:51:13.391386 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:13.391388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:13.391558 (+ 170us) server_negotiation.cc:407] Connection header received
0926 22:51:13.392039 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:13.392041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:13.392055 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:13.392145 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:13.392660 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.393352 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.403367 (+ 10015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.403673 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.403750 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:13.406491 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:13.406493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:13.406493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:13.406501 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:13.408095 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:13.408579 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:13.408580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:13.408581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:13.408640 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:13.408927 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:13.408928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:13.408928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:13.409099 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:13.409142 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:13.409581 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 22:51:13.409666 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:51:13.410660 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:13.391266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35853 (local address 127.13.148.3:51589)
0926 22:51:13.391426 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:13.391475 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:13.391947 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:13.392198 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:13.392202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:13.392375 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:13.392577 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:13.392583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.401876 (+ 9293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.401878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:13.403264 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:13.403269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.403838 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.403840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:13.404041 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:13.404663 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:13.404674 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:13.406377 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:13.408191 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:13.408198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:13.408200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:13.408489 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:13.408713 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:13.408714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:13.408714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:13.408849 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:13.409258 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:13.409265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:13.409427 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 22:51:13.410265 (+ 838us) client_negotiation.cc:240] Negotiation successful
0926 22:51:13.410429 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260926 22:51:13.411298 26590 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35853
I20260926 22:51:13.411417 26590 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:13.411604 26590 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:13.411952 26106 ts_manager.cc:194] Registered new tserver with Master: 196a3d2593234026bfb72720852dab8c (127.13.148.3:37341)
I20260926 22:51:13.412439 26106 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:51589
I20260926 22:51:13.419982 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:51:13.420871 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 26187
I20260926 22:51:13.427302 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 26324
I20260926 22:51:13.433600 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 26461
I20260926 22:51:13.438706 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 26074
I20260926 22:51:13.443563 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35853
--webserver_interface=127.13.148.62
--webserver_port=33267
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35853
--ranger_config_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:35449/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:35449/kms with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:13.521142 26593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:13.521348 26593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:13.521377 26593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:13.522504 26593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:13.522545 26593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:13.522563 26593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:13.522579 26593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:13.522593 26593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:13.524122 26593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:35449/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35853
--ranger_config_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=33267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:13.524367 26593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:13.524562 26593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:13.528970 26598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:13.528983 26599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:13.528980 26601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:13.529060 26593 server_base.cc:1034] running on GCE node
I20260926 22:51:13.529400 26593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:13.529670 26593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:13.530799 26593 hybrid_clock.cc:648] HybridClock initialized: now 1790463073530793 us; error 42 us; skew 500 ppm
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:13.533016 26593 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:51:13.533422 26593 webserver.cc:460] Webserver started at http://127.13.148.62:33267/ using document root <none> and password file <none>
I20260926 22:51:13.533614 26593 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:13.533676 26593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:13.587728 26593 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:13.588505 26609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.588732 26593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:13.588833 26593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "257ac573577d40d48832c88eae17236f"
format_stamp: "Formatted at 2026-09-26 22:51:11 on dist-test-slave-98hb"
server_key: "9474384dad981825483281259cf7fb89"
server_key_iv: "52a1df7d0f05a82baac2c3adc0df493d"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.589300 26593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:13.606493 26593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:13.608637 26593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:13.608796 26593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:13.613533 26593 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35853
I20260926 22:51:13.613579 26661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35853 every 8 connection(s)
I20260926 22:51:13.614029 26593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:51:13.615243 26662 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:51:13.615895 26662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Bootstrap starting.
I20260926 22:51:13.618840 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 26593
I20260926 22:51:13.619146 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:35449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:46473
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:35853
--webserver_port=41415
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:35449/kms with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:13.624022 26662 log.cc:824] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:13.625586 26662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:51:13.625885 26662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Bootstrap complete.
I20260926 22:51:13.628023 26662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } }
I20260926 22:51:13.628258 26662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257ac573577d40d48832c88eae17236f, State: Initialized, Role: FOLLOWER
I20260926 22:51:13.628440 26662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } }
I20260926 22:51:13.628515 26662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:51:13.628575 26662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:51:13.628628 26662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:51:13.629585 26662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } }
I20260926 22:51:13.629679 26662 leader_election.cc:302] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 257ac573577d40d48832c88eae17236f; no voters:
I20260926 22:51:13.629873 26662 leader_election.cc:288] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 22:51:13.629916 26666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:51:13.630098 26666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [term 2 LEADER]: Becoming Leader. State: Replica: 257ac573577d40d48832c88eae17236f, State: Running, Role: LEADER
I20260926 22:51:13.630168 26662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:13.630255 26666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } }
I20260926 22:51:13.630578 26666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "257ac573577d40d48832c88eae17236f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } } }
I20260926 22:51:13.630728 26666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:13.630891 26666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 257ac573577d40d48832c88eae17236f. Latest consensus state: current_term: 2 leader_uuid: "257ac573577d40d48832c88eae17236f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257ac573577d40d48832c88eae17236f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35853 } } }
I20260926 22:51:13.630986 26666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:13.631543 26662 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 22:51:13.631685 26675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:51:13.632103 26675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:51:13.632239 26675 catalog_manager.cc:1259] Loaded cluster ID: 048d60a4cff14cc58f13ad29a350f0ec
I20260926 22:51:13.632269 26675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:51:13.633072 26675 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:51:13.633226 26675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Loaded TSK: 0
I20260926 22:51:13.633409 26675 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:51:13.732623 26664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:13.732892 26664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:13.732931 26664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:13.734689 26664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:13.734776 26664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:13.734833 26664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:51:13.736939 26664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:35449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:46473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=41415
--tserver_master_addrs=127.13.148.62:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:13.737311 26664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:13.737612 26664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:13.743934 26699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:13.744059 26700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:13.743930 26702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:13.744895 26664 server_base.cc:1034] running on GCE node
I20260926 22:51:13.745121 26664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:13.745389 26664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:13.746558 26664 hybrid_clock.cc:648] HybridClock initialized: now 1790463073746539 us; error 33 us; skew 500 ppm
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:13.750144 26664 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:51:13.750626 26664 webserver.cc:460] Webserver started at http://127.13.148.1:41415/ using document root <none> and password file <none>
I20260926 22:51:13.750859 26664 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:13.750918 26664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:13.776904 26664 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:13.778167 26710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.778357 26664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:13.778441 26664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "de23bcce36fa4bc7a022dfd68112762f"
format_stamp: "Formatted at 2026-09-26 22:51:12 on dist-test-slave-98hb"
server_key: "adb9b3c24253d8479aca9019d9747e63"
server_key_iv: "5e6eeb37e31e2de3a780ed2c17b9df0e"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.778891 26664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:13.794145 26664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:13.796525 26664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:13.796748 26664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:13.797003 26664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:13.797410 26664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:13.797498 26664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.797610 26664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:13.797684 26664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.803612 26664 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:46473
I20260926 22:51:13.804140 26664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:51:13.806177 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 26664
I20260926 22:51:13.806439 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:35449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:46609
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:35853
--webserver_port=45597
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:35449/kms with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:13.809387 26823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:46473 every 8 connection(s)
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:13.825495 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:13.805426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:52425 (local address 127.13.148.62:35853)
0926 22:51:13.814888 (+ 9462us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:13.814896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:13.814928 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:51:13.815064 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:13.815070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:13.815125 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:13.815242 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:13.816029 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.817404 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.818883 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.819464 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.819555 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:13.822013 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:13.822018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:13.822018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:13.822029 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:13.823807 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:13.824256 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:13.824259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:13.824259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:13.824351 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:13.824594 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:13.824595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:13.824596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:13.824846 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:13.824915 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:13.825202 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 22:51:13.825303 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9409,"thread_start_us":77,"threads_started":1}
I20260926 22:51:13.825505 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:13.805080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35853 (local address 127.13.148.1:52425)
0926 22:51:13.813887 (+ 8807us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:13.813941 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:13.814551 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:13.815413 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:13.815416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:13.815642 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:13.815894 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:13.815899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.817519 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.817520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:13.818794 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:13.818797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:13.819667 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:13.819668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:13.819791 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:13.820325 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:13.820335 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:13.821906 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:13.823896 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:13.823902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:13.823904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:13.824166 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:13.824425 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:13.824425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:13.824426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:13.824525 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:13.824909 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:13.824915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:13.825041 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 22:51:13.825128 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 22:51:13.825282 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8756,"thread_start_us":75,"threads_started":1}
I20260926 22:51:13.826645 26824 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35853
I20260926 22:51:13.826742 26824 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:13.826913 26824 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
W20260926 22:51:13.827649 26824 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:35853 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 22:51:13.827710 26824 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:13.827777 26824 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:13.828051 26824 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:13.828122 26824 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
W20260926 22:51:13.828383 26824 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:51:13.933517 26828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:13.933881 26828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:13.933985 26828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:13.935153 26828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:13.935191 26828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:13.935227 26828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:51:13.936844 26828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:35449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:46609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=45597
--tserver_master_addrs=127.13.148.62:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:13.937068 26828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:13.937299 26828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:13.942121 26836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:13.942171 26833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:13.942315 26828 server_base.cc:1034] running on GCE node
W20260926 22:51:13.942469 26834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:13.942656 26828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:13.942899 26828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:13.944049 26828 hybrid_clock.cc:648] HybridClock initialized: now 1790463073944036 us; error 33 us; skew 500 ppm
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:13.947535 26828 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:51:13.947933 26828 webserver.cc:460] Webserver started at http://127.13.148.2:45597/ using document root <none> and password file <none>
I20260926 22:51:13.948127 26828 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:13.948186 26828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 22:51:13 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:13.973277 26828 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260926 22:51:13.974301 26844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.974500 26828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:13.974597 26828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "4601464d35194aa5b2f8cc7bb7dfacfb"
format_stamp: "Formatted at 2026-09-26 22:51:13 on dist-test-slave-98hb"
server_key: "905d8077c2091b40650ab7b731f47a28"
server_key_iv: "1a145672e0e9bbcb7725965bae8d4221"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:13.974977 26828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:13.995738 26828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:13.997572 26828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:13.997710 26828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:13.997991 26828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:13.998327 26828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:13.998363 26828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:13.998391 26828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:13.998414 26828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:14.004732 26828 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:46609
I20260926 22:51:14.005039 26957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:46609 every 8 connection(s)
I20260926 22:51:14.005107 26828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:51:14 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463073, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:14.013705 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 26828
I20260926 22:51:14.014008 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.148.20:35449/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:37341
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:35853
--webserver_port=35477
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:35449/kms with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:14.015092 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:14.005895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:38685 (local address 127.13.148.62:35853)
0926 22:51:14.005939 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:14.005941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:14.006264 (+ 323us) server_negotiation.cc:407] Connection header received
0926 22:51:14.006765 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:14.006766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:14.006782 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:14.006832 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:14.007403 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.007900 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.008874 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.009079 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.009134 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:14.011708 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:14.011710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:14.011710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:14.011718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:14.013397 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:14.013958 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:14.013959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:14.013960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:14.014024 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:14.014338 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:14.014339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:14.014339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:14.014482 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:14.014526 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:14.014917 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 22:51:14.015004 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:51:14.015290 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:14.005935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35853 (local address 127.13.148.2:38685)
0926 22:51:14.006108 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:14.006156 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:14.006648 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:14.006941 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:14.006945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:14.007108 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:14.007310 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:14.007315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.007983 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.007984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:14.008791 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:14.008794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.009175 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.009176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:14.009310 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:14.009924 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:14.009934 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:14.011591 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:14.013492 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:14.013499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:14.013502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:14.013851 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:14.014094 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:14.014095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:14.014095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:14.014250 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:14.014572 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:14.014579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:14.014753 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 22:51:14.014877 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 22:51:14.015037 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 22:51:14.015739 26958 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35853
I20260926 22:51:14.015836 26958 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:14.016049 26958 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
W20260926 22:51:14.016707 26958 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:35853 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 22:51:14.016780 26958 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:14.016844 26958 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:14.017206 26958 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:14.017278 26958 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
W20260926 22:51:14.017540 26958 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:51:14.130314 26961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:14.130504 26961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:14.130532 26961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:14.131809 26961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:14.131853 26961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:14.131879 26961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:51:14.133296 26961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.148.20:35449/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:37341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=35477
--tserver_master_addrs=127.13.148.62:35853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:14.133497 26961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:14.133771 26961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:14.139762 26968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:14.139858 26970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:14.140059 26967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:14.140122 26961 server_base.cc:1034] running on GCE node
I20260926 22:51:14.140303 26961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:14.140555 26961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:14.141691 26961 hybrid_clock.cc:648] HybridClock initialized: now 1790463074141672 us; error 37 us; skew 500 ppm
Sep 26 22:51:14 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:14.144313 26961 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:51:14.144824 26961 webserver.cc:460] Webserver started at http://127.13.148.3:35477/ using document root <none> and password file <none>
I20260926 22:51:14.145084 26961 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:14.145169 26961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 22:51:14 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:14.162703 26961 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:14.163709 26978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:14.163882 26961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:51:14.163944 26961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "196a3d2593234026bfb72720852dab8c"
format_stamp: "Formatted at 2026-09-26 22:51:13 on dist-test-slave-98hb"
server_key: "707dbcf3bd48f791249a0007a68ae1b0"
server_key_iv: "c060ed8b000142bef684a6b0cf18db5d"
server_key_version: "kuduclusterkey@0"
I20260926 22:51:14.164393 26961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:14.185811 26961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:14.188076 26961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:14.188249 26961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:14.188552 26961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:14.188906 26961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:14.188956 26961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:14.188985 26961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:14.189043 26961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:14.196102 26961 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:37341
I20260926 22:51:14.196280 27091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:37341 every 8 connection(s)
I20260926 22:51:14.196635 26961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:51:14.200218 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 26961
I20260926 22:51:14.208689 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:14.200837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.13.148.62:35853)
0926 22:51:14.200892 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:14.200895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:14.201074 (+ 179us) server_negotiation.cc:407] Connection header received
0926 22:51:14.201202 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:14.201203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:14.201216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:14.201247 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:14.202236 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.202969 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.204043 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.204336 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.204383 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:14.205953 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:14.205954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:14.205954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:14.205961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:14.207534 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:14.207916 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:14.207917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:14.207917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:14.207980 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:14.208174 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:14.208175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:14.208175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:14.208302 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:14.208469 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:14.208510 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 22:51:14.208597 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 22:51:14 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:14.215819 27095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:14.206831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:51331 (local address 127.13.148.62:35853)
0926 22:51:14.207645 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:14.207647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:14.207653 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:51:14.207907 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:14.208045 (+ 138us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:14.208059 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:14.208116 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:14.208816 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.209499 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.210954 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.211152 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.211207 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:14.213227 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:14.213228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:14.213228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:14.213234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:14.214607 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:14.214966 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:14.214967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:14.214967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:14.215017 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:14.215232 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:14.215233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:14.215233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:14.215339 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:14.215400 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:14.215679 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 22:51:14.215743 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":64,"threads_started":1}
I20260926 22:51:14.216071 27096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:14.207049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:35853 (local address 127.13.148.3:51331)
0926 22:51:14.207212 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:14.207257 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:14.207781 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:14.208229 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:14.208234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:14.208423 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:14.208730 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:14.208737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.209569 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.209570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:14.210868 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:14.210871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:14.211250 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:14.211251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:14.211397 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:14.211842 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:14.211852 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:14.213141 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:14.214684 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:14.214690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:14.214693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:14.214900 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:14.215068 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:14.215069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:14.215069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:14.215176 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:14.215406 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:14.215413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:14.215545 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 22:51:14.215656 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 22:51:14.215816 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260926 22:51:14.216540 27092 heartbeater.cc:347] Connected to a master server at 127.13.148.62:35853
I20260926 22:51:14.216631 27092 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:14.216801 27092 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
W20260926 22:51:14.217361 27092 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:35853 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 22:51:14.217417 27092 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:14.217465 27092 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:14.217796 27092 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:14.217872 27092 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
W20260926 22:51:14.218144 27092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 22:51:14 dist-test-slave-98hb krb5kdc[25630](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790463074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:14.828564 26824 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:14.828735 26824 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:14.829136 26626 ts_manager.cc:194] Registered new tserver with Master: de23bcce36fa4bc7a022dfd68112762f (127.13.148.1:46473)
I20260926 22:51:14.830224 26626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:52425
I20260926 22:51:15.017702 26958 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:15.017880 26958 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:15.018146 26626 ts_manager.cc:194] Registered new tserver with Master: 4601464d35194aa5b2f8cc7bb7dfacfb (127.13.148.2:46609)
I20260926 22:51:15.018671 26626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:38685
Sep 26 22:51:15 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
I20260926 22:51:15.218315 27092 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:15.218477 27092 heartbeater.cc:511] Master 127.13.148.62:35853 requested a full tablet report, sending...
I20260926 22:51:15.218837 26626 ts_manager.cc:194] Registered new tserver with Master: 196a3d2593234026bfb72720852dab8c (127.13.148.3:37341)
I20260926 22:51:15.219367 26626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:51331
I20260926 22:51:15.221978 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:51:15.230657 27114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.222893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.13.148.62:35853)
0926 22:51:15.223140 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:15.223142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:15.223149 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:15.223170 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:15.223170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:15.223182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:15.223243 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:15.223863 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.224645 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.225600 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.225948 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.226015 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.227478 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:15.227479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:15.227480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:15.227487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:15.229146 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:15.229558 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:15.229559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:15.229560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:15.229628 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:15.229888 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:15.229889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:15.229889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:15.230080 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:15.230178 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:15.230396 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 22:51:15.230540 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":70,"threads_started":1}
I20260926 22:51:15.232736 26626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:51:15.235136 26626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:51:15.244577 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.239754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.13.148.1:46473)
0926 22:51:15.239970 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:15.239975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:15.240477 (+ 502us) server_negotiation.cc:407] Connection header received
0926 22:51:15.240606 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:15.240608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:15.240624 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:15.240670 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:15.240912 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.241952 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.243372 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.244089 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.244134 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.244201 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:15.244454 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 22:51:15.244495 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":57,"threads_started":1}
I20260926 22:51:15.245024 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.239246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46473 (local address 127.0.0.1:59668)
0926 22:51:15.240368 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:15.240410 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:15.240515 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:15.240778 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:15.240781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:15.240783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:15.240847 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.240849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.242042 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.242043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.243275 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.243278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.244266 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.244267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.244344 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.244356 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 22:51:15.244888 (+ 532us) client_negotiation.cc:240] Negotiation successful
0926 22:51:15.244953 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":94,"threads_started":1}
I20260926 22:51:15.245934 26758 tablet_service.cc:1467] Processing CreateTablet for tablet 9c8406ed7c9d4298a2b6771a76201a20 (DEFAULT_TABLE table=test-table [id=586fff7adcc045f9a20ac281f01a2566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:15.246305 26758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c8406ed7c9d4298a2b6771a76201a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:15.249430 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.239901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.13.148.3:37341)
0926 22:51:15.240051 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:15.240054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:15.240688 (+ 634us) server_negotiation.cc:407] Connection header received
0926 22:51:15.240793 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:15.240796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:15.240811 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:15.240863 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:15.241448 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.242821 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.248211 (+ 5390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.248972 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.249019 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.249089 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:15.249313 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 22:51:15.249354 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":54,"threads_started":1}
I20260926 22:51:15.249655 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.239488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:37341 (local address 127.0.0.1:45084)
0926 22:51:15.240622 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:15.240632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:15.240716 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:15.240963 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:15.240964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:15.240966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:15.241024 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.241026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.242917 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.242917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.244035 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.244038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.249154 (+ 5116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.249154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.249228 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.249231 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:51:15.249537 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 22:51:15.249586 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"thread_start_us":66,"threads_started":1}
I20260926 22:51:15.250295 27125 tablet_bootstrap.cc:492] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f: Bootstrap starting.
I20260926 22:51:15.250344 27026 tablet_service.cc:1467] Processing CreateTablet for tablet 9c8406ed7c9d4298a2b6771a76201a20 (DEFAULT_TABLE table=test-table [id=586fff7adcc045f9a20ac281f01a2566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:15.250720 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c8406ed7c9d4298a2b6771a76201a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:15.250958 27125 tablet_bootstrap.cc:654] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:15.251240 27125 log.cc:824] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:15.251919 27125 tablet_bootstrap.cc:492] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f: No bootstrap required, opened a new log
I20260926 22:51:15.251972 27125 ts_tablet_manager.cc:1397] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:15.254419 27125 raft_consensus.cc:348] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.254585 27125 raft_consensus.cc:374] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:15.254617 27125 raft_consensus.cc:729] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de23bcce36fa4bc7a022dfd68112762f, State: Initialized, Role: FOLLOWER
I20260926 22:51:15.254870 27125 consensus_queue.cc:260] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.255064 27120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.240119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46609 (local address 127.0.0.1:50776)
0926 22:51:15.241113 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:15.241121 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:15.241164 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:15.244129 (+ 2965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:15.244130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:15.244132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:15.244191 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.244192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.245706 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.245707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.246823 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.246826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.254799 (+ 7973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.254800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.254885 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.254890 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:51:15.254942 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:51:15.254987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":72,"threads_started":1}
I20260926 22:51:15.255352 27125 ts_tablet_manager.cc:1428] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:51:15.255364 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.240214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.13.148.2:46609)
0926 22:51:15.240552 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:15.240557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:15.241216 (+ 659us) server_negotiation.cc:407] Connection header received
0926 22:51:15.241254 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:15.241256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:15.241270 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:15.241337 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:15.244266 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.245610 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.246917 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.247990 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.248081 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.248146 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:15.255229 (+ 7083us) server_negotiation.cc:299] Negotiation successful
0926 22:51:15.255287 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":62,"threads_started":1}
I20260926 22:51:15.255767 27127 tablet_bootstrap.cc:492] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Bootstrap starting.
I20260926 22:51:15.256067 26892 tablet_service.cc:1467] Processing CreateTablet for tablet 9c8406ed7c9d4298a2b6771a76201a20 (DEFAULT_TABLE table=test-table [id=586fff7adcc045f9a20ac281f01a2566]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:15.256426 26892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c8406ed7c9d4298a2b6771a76201a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:15.256451 27127 tablet_bootstrap.cc:654] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:15.256727 27127 log.cc:824] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:15.257476 27127 tablet_bootstrap.cc:492] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: No bootstrap required, opened a new log
I20260926 22:51:15.257534 27127 ts_tablet_manager.cc:1397] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:15.260008 27127 raft_consensus.cc:348] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.260195 27127 raft_consensus.cc:374] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:15.260227 27127 raft_consensus.cc:729] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196a3d2593234026bfb72720852dab8c, State: Initialized, Role: FOLLOWER
I20260926 22:51:15.255611 26824 heartbeater.cc:503] Master 127.13.148.62:35853 was elected leader, sending a full tablet report...
I20260926 22:51:15.260339 27127 consensus_queue.cc:260] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.260574 27127 ts_tablet_manager.cc:1428] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:51:15.260977 27092 heartbeater.cc:503] Master 127.13.148.62:35853 was elected leader, sending a full tablet report...
I20260926 22:51:15.261938 27130 tablet_bootstrap.cc:492] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Bootstrap starting.
I20260926 22:51:15.262612 27130 tablet_bootstrap.cc:654] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:15.262917 27130 log.cc:824] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:15.263733 27130 tablet_bootstrap.cc:492] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: No bootstrap required, opened a new log
I20260926 22:51:15.263809 27130 ts_tablet_manager.cc:1397] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:15.266570 27130 raft_consensus.cc:348] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.266776 27130 raft_consensus.cc:374] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:15.266835 27130 raft_consensus.cc:729] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4601464d35194aa5b2f8cc7bb7dfacfb, State: Initialized, Role: FOLLOWER
I20260926 22:51:15.266994 27130 consensus_queue.cc:260] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.267305 27130 ts_tablet_manager.cc:1428] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 22:51:15.267441 26958 heartbeater.cc:503] Master 127.13.148.62:35853 was elected leader, sending a full tablet report...
Sep 26 22:51:15 dist-test-slave-98hb krb5kdc[25630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463074, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for HTTP/127.13.148.20@KRBTEST.COM
W20260926 22:51:15.315702 26825 tablet.cc:2369] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:51:15.457453 27093 tablet.cc:2369] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:51:15.506505 26959 tablet.cc:2369] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:51:15.532883 27133 raft_consensus.cc:482] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:51:15.533010 27133 raft_consensus.cc:504] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.533368 27133 leader_election.cc:288] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de23bcce36fa4bc7a022dfd68112762f (127.13.148.1:46473), 196a3d2593234026bfb72720852dab8c (127.13.148.3:37341)
I20260926 22:51:15.537249 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.533755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:53235 (local address 127.13.148.3:37341)
0926 22:51:15.533794 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:15.533796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:15.533896 (+ 100us) server_negotiation.cc:407] Connection header received
0926 22:51:15.533986 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:15.533987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:15.533996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:15.534044 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:15.534189 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.534861 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.536346 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.536917 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.536989 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.536998 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:15.537156 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 22:51:15.537198 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:51:15.537281 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.533541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:37341 (local address 127.13.148.2:53235)
0926 22:51:15.533827 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:15.533836 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:15.533895 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:15.534048 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:15.534049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:15.534051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:15.534124 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.534126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.534926 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.534927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.536258 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.536260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.537006 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.537007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.537077 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.537082 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:51:15.537141 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:51:15.537184 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":60,"thread_start_us":81,"threads_started":1}
I20260926 22:51:15.537297 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.533587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:56101 (local address 127.13.148.1:46473)
0926 22:51:15.533756 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:15.533759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:15.533768 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:51:15.533861 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:15.533862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:15.533871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:15.533917 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:15.534104 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.534851 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.536345 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.536916 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.536989 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.536998 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:15.537192 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 22:51:15.537243 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 22:51:15.537467 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.533511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46473 (local address 127.13.148.2:56101)
0926 22:51:15.533662 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:15.533672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:15.533725 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:15.533933 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:15.533934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:15.533936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:15.534013 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.534015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.534923 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.534925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.536262 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:15.536264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.537032 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.537033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:15.537107 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.537109 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:51:15.537358 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 22:51:15.537395 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":70,"thread_start_us":94,"threads_started":1}
I20260926 22:51:15.537629 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c8406ed7c9d4298a2b6771a76201a20" candidate_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196a3d2593234026bfb72720852dab8c" is_pre_election: true
I20260926 22:51:15.537799 27046 raft_consensus.cc:2454] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4601464d35194aa5b2f8cc7bb7dfacfb in term 0.
I20260926 22:51:15.537863 26770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c8406ed7c9d4298a2b6771a76201a20" candidate_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de23bcce36fa4bc7a022dfd68112762f" is_pre_election: true
I20260926 22:51:15.538020 26770 raft_consensus.cc:2454] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4601464d35194aa5b2f8cc7bb7dfacfb in term 0.
I20260926 22:51:15.538036 26847 leader_election.cc:302] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 196a3d2593234026bfb72720852dab8c, 4601464d35194aa5b2f8cc7bb7dfacfb; no voters:
I20260926 22:51:15.538151 27133 raft_consensus.cc:2781] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:51:15.538223 27133 raft_consensus.cc:482] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:51:15.538244 27133 raft_consensus.cc:3037] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:15.538861 27133 raft_consensus.cc:504] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.538970 27133 leader_election.cc:288] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [CANDIDATE]: Term 1 election: Requested vote from peers de23bcce36fa4bc7a022dfd68112762f (127.13.148.1:46473), 196a3d2593234026bfb72720852dab8c (127.13.148.3:37341)
I20260926 22:51:15.539122 26770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c8406ed7c9d4298a2b6771a76201a20" candidate_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de23bcce36fa4bc7a022dfd68112762f"
I20260926 22:51:15.539186 26770 raft_consensus.cc:3037] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:15.539855 26770 raft_consensus.cc:2454] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4601464d35194aa5b2f8cc7bb7dfacfb in term 1.
I20260926 22:51:15.539966 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c8406ed7c9d4298a2b6771a76201a20" candidate_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "196a3d2593234026bfb72720852dab8c"
I20260926 22:51:15.540009 26848 leader_election.cc:302] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4601464d35194aa5b2f8cc7bb7dfacfb, de23bcce36fa4bc7a022dfd68112762f; no voters:
I20260926 22:51:15.540040 27046 raft_consensus.cc:3037] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:15.540083 27133 raft_consensus.cc:2781] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:15.540203 27133 raft_consensus.cc:686] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 1 LEADER]: Becoming Leader. State: Replica: 4601464d35194aa5b2f8cc7bb7dfacfb, State: Running, Role: LEADER
I20260926 22:51:15.540288 27133 consensus_queue.cc:237] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } }
I20260926 22:51:15.540838 27046 raft_consensus.cc:2454] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4601464d35194aa5b2f8cc7bb7dfacfb in term 1.
I20260926 22:51:15.541182 26625 catalog_manager.cc:5495] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 4601464d35194aa5b2f8cc7bb7dfacfb (127.13.148.2). New cstate: current_term: 1 leader_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4601464d35194aa5b2f8cc7bb7dfacfb" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46609 } health_report { overall_health: HEALTHY } } }
I20260926 22:51:15.601157 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.598919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.13.148.2:46609)
0926 22:51:15.598974 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:15.598976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:15.598983 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:15.599023 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:15.599024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:15.599041 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:15.599099 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:51:15.599241 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.599708 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.600446 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:15.600636 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:15.600692 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:15.600850 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:51:15.600950 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:51:15.601022 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:15.601057 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 22:51:15.601077 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:51:15.602945 26770 raft_consensus.cc:1261] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 1 FOLLOWER]: Refusing update from remote peer 4601464d35194aa5b2f8cc7bb7dfacfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:15.602941 27046 raft_consensus.cc:1261] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 1 FOLLOWER]: Refusing update from remote peer 4601464d35194aa5b2f8cc7bb7dfacfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:15.603310 27145 consensus_queue.cc:1035] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "196a3d2593234026bfb72720852dab8c" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 37341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:15.603452 27133 consensus_queue.cc:1035] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "de23bcce36fa4bc7a022dfd68112762f" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 46473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:15.606428 27150 mvcc.cc:205] Tried to move back new op lower bound from 7333736757666996224 to 7333736757413658624. Current Snapshot: MvccSnapshot[applied={T|T < 7333736757666996224}]
I20260926 22:51:15.608021 26626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:51:15.608101 26626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:51:15.608743 26626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 257ac573577d40d48832c88eae17236f: Sending DeleteTablet for 3 replicas of tablet 9c8406ed7c9d4298a2b6771a76201a20
I20260926 22:51:15.609030 26758 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c8406ed7c9d4298a2b6771a76201a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:59668
I20260926 22:51:15.609050 26892 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c8406ed7c9d4298a2b6771a76201a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:50776
I20260926 22:51:15.609033 27026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c8406ed7c9d4298a2b6771a76201a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:15 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:45084
I20260926 22:51:15.609246 27157 tablet_replica.cc:330] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f: stopping tablet replica
I20260926 22:51:15.609340 27157 raft_consensus.cc:2229] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:51:15.609395 27159 tablet_replica.cc:330] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: stopping tablet replica
I20260926 22:51:15.609462 27157 raft_consensus.cc:2258] T 9c8406ed7c9d4298a2b6771a76201a20 P de23bcce36fa4bc7a022dfd68112762f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:15.609516 27159 raft_consensus.cc:2229] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:51:15.609555 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 26664
I20260926 22:51:15.609628 27159 raft_consensus.cc:2258] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:15.610042 27159 ts_tablet_manager.cc:1905] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:15.611776 27158 tablet_replica.cc:330] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: stopping tablet replica
I20260926 22:51:15.611865 27158 raft_consensus.cc:2229] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:51:15.611887 27159 ts_tablet_manager.cc:1918] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:51:15.611948 27159 log.cc:1196] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/9c8406ed7c9d4298a2b6771a76201a20
I20260926 22:51:15.611969 27158 raft_consensus.cc:2258] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:15.612284 27159 ts_tablet_manager.cc:1939] T 9c8406ed7c9d4298a2b6771a76201a20 P 196a3d2593234026bfb72720852dab8c: Deleting consensus metadata
I20260926 22:51:15.612388 27158 ts_tablet_manager.cc:1905] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:15.612665 26612 catalog_manager.cc:4841] TS 196a3d2593234026bfb72720852dab8c (127.13.148.3:37341): tablet 9c8406ed7c9d4298a2b6771a76201a20 (table test-table [id=586fff7adcc045f9a20ac281f01a2566]) successfully deleted
I20260926 22:51:15.614152 27158 ts_tablet_manager.cc:1918] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:51:15.614283 27158 log.cc:1196] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/9c8406ed7c9d4298a2b6771a76201a20
I20260926 22:51:15.614578 27158 ts_tablet_manager.cc:1939] T 9c8406ed7c9d4298a2b6771a76201a20 P 4601464d35194aa5b2f8cc7bb7dfacfb: Deleting consensus metadata
I20260926 22:51:15.615455 26610 catalog_manager.cc:4841] TS 4601464d35194aa5b2f8cc7bb7dfacfb (127.13.148.2:46609): tablet 9c8406ed7c9d4298a2b6771a76201a20 (table test-table [id=586fff7adcc045f9a20ac281f01a2566]) successfully deleted
W20260926 22:51:15.616007 26613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:46473 (error 108)
I20260926 22:51:15.616319 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 26828
I20260926 22:51:15.616912 27120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:15.616368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:46473 (local address 127.0.0.1:59676)
0926 22:51:15.616845 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:15.616882 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:46473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458}
W20260926 22:51:15.617092 26613 catalog_manager.cc:4584] TS de23bcce36fa4bc7a022dfd68112762f (127.13.148.1:46473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c8406ed7c9d4298a2b6771a76201a20: Network error: Client connection negotiation failed: client connection to 127.13.148.1:46473: connect: Connection refused (error 111)
I20260926 22:51:15.620676 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 26961
I20260926 22:51:15.624378 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 26593
I20260926 22:51:15.630957 13904 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 22:51:15.964545 13904 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 22:51:15.964638 13904 mini_ranger.cc:66] Stopping Ranger...
I20260926 22:51:16.298233 13904 mini_ranger.cc:68] Stopped Ranger
2026-09-26 22:51:16.298 UTC [25660] LOG: received smart shutdown request
2026-09-26 22:51:16.302 UTC [25660] LOG: background worker "logical replication launcher" (PID 25669) exited with exit code 1
2026-09-26 22:51:16.302 UTC [25664] LOG: shutting down
2026-09-26 22:51:16.436 UTC [25660] LOG: database system is shut down
2026-09-26T22:51:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46368 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:51:16 dist-test-slave-98hb krb5kdc[27169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:51:16 dist-test-slave-98hb krb5kdc[27169](info): set up 2 sockets
Sep 26 22:51:16 dist-test-slave-98hb krb5kdc[27169](info): commencing operation
krb5kdc: starting...
W20260926 22:51:17.461760 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:51:17.570147 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36763
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36763,127.13.148.61:37933,127.13.148.60:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:17.647709 27182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:17.647905 27182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:17.647925 27182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:17.648972 27182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:17.649003 27182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:17.649015 27182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:17.649025 27182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:17.649040 27182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:17.650486 27182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36763,127.13.148.61:37933,127.13.148.60:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:17.650725 27182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:17.650967 27182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:17.655444 27188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:17.655457 27187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:17.655548 27182 server_base.cc:1034] running on GCE node
W20260926 22:51:17.655484 27190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:17.655813 27182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:17.656086 27182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:17.657215 27182 hybrid_clock.cc:648] HybridClock initialized: now 1790463077657203 us; error 29 us; skew 500 ppm
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:17.664000 27182 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:51:17.664402 27182 webserver.cc:460] Webserver started at http://127.13.148.62:35445/ using document root <none> and password file <none>
I20260926 22:51:17.664575 27182 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:17.664613 27182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:17.664702 27182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:17.665642 27182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "5beb4a00aca043819746f9af5bdd208f"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "57df80a37912291609aba7fc597a3973"
server_key_iv: "ac31f28220624cfd12e5b9aae31bb19c"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.665978 27182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "5beb4a00aca043819746f9af5bdd208f"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "57df80a37912291609aba7fc597a3973"
server_key_iv: "ac31f28220624cfd12e5b9aae31bb19c"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.667083 27182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:17.667688 27197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:17.667829 27182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:51:17.667912 27182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "5beb4a00aca043819746f9af5bdd208f"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "57df80a37912291609aba7fc597a3973"
server_key_iv: "ac31f28220624cfd12e5b9aae31bb19c"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.667964 27182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:17.677006 27182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:17.678984 27182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:17.679090 27182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:17.682739 27182 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36763
I20260926 22:51:17.682780 27249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36763 every 8 connection(s)
I20260926 22:51:17.683104 27182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:51:17.683939 27250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:17.683935 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 27182
I20260926 22:51:17.684015 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:51:17.684201 13904 external_mini_cluster.cc:1428] Setting key 7df5aa895338033c23818dd673501359
I20260926 22:51:17.685616 27250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.61@KRBTEST.COM" created.
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:17.697968 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.686606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.13.148.62:36763)
0926 22:51:17.686734 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:17.686738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:17.686770 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:51:17.687235 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:17.687237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:17.687258 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:17.687333 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:17.687914 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.688726 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.689573 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.689945 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.690049 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.694850 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:17.694857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:17.694858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:17.694864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:17.696507 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.696846 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.696850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.696850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.696930 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.697113 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.697113 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.697114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.697334 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:17.697404 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:17.697575 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 22:51:17.697692 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260926 22:51:17.697970 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.686420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36763 (local address 127.0.0.1:40210)
0926 22:51:17.686574 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.686608 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:17.687090 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:17.687343 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:17.687346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:17.687662 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:17.687819 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.687824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.688811 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.688812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.689501 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.689503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.690017 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.690018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.690083 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.690593 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:17.690601 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:17.694763 (+ 4162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:17.696588 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.696594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.696596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.696777 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.696984 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.696985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.696985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.697060 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.697392 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:17.697397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:17.697459 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 22:51:17.697537 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:51:17.697784 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":84,"thread_start_us":75,"threads_started":1}
I20260926 22:51:17.698652 27250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:51:17.698989 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.698882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:47222)
0926 22:51:17.698924 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.698966 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 22:51:17.699106 27201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
I20260926 22:51:17.699460 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.699379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:47228)
0926 22:51:17.699414 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.699437 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 22:51:17.699553 27250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:37933: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
I20260926 22:51:17.706801 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:37933
--webserver_interface=127.13.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36763,127.13.148.61:37933,127.13.148.60:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:17.748773 27250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } attempt: 1
I20260926 22:51:17.749291 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.749159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:49530)
0926 22:51:17.749220 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.749261 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 22:51:17.749606 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.749523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:49540)
0926 22:51:17.749560 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.749584 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 22:51:17.749707 27250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:37933: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
W20260926 22:51:17.786005 27257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:17.786196 27257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:17.786218 27257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:17.787295 27257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:17.787326 27257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:17.787338 27257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:17.787349 27257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:17.787359 27257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:17.788800 27257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36763,127.13.148.61:37933,127.13.148.60:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:37933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:17.788977 27257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:17.789144 27257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:17.793570 27263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:17.793555 27265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:17.793597 27262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:17.793828 27257 server_base.cc:1034] running on GCE node
I20260926 22:51:17.794076 27257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:17.794340 27257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:17.795499 27257 hybrid_clock.cc:648] HybridClock initialized: now 1790463077795482 us; error 29 us; skew 500 ppm
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:17.797791 27257 init.cc:377] Logged in from keytab as kudu/127.13.148.61@KRBTEST.COM (short username kudu)
I20260926 22:51:17.798173 27257 webserver.cc:460] Webserver started at http://127.13.148.61:42693/ using document root <none> and password file <none>
I20260926 22:51:17.798357 27257 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:17.798395 27257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:17.798484 27257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:17.799448 27257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/data/instance:
uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "bd85c3837b0462333cfc267def8cc13c"
server_key_iv: "c05b7e9f3a0babfec449da386f525a19"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.799727 27257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/wal/instance:
uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "bd85c3837b0462333cfc267def8cc13c"
server_key_iv: "c05b7e9f3a0babfec449da386f525a19"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.800859 27257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:51:17.801468 27272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:17.801599 27257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:17.801664 27257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/wal
uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "bd85c3837b0462333cfc267def8cc13c"
server_key_iv: "c05b7e9f3a0babfec449da386f525a19"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.801713 27257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:17.817886 27250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } attempt: 2
I20260926 22:51:17.818325 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.818199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:49556)
0926 22:51:17.818255 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.818296 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:51:17.818684 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.818580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:49558)
0926 22:51:17.818622 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.818647 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 22:51:17.818791 27250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:37933: Network error: Client connection negotiation failed: client connection to 127.13.148.61:37933: connect: Connection refused (error 111)
I20260926 22:51:17.826167 27257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:17.827885 27257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:17.828012 27257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:17.831691 27257 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:37933
I20260926 22:51:17.831722 27324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:37933 every 8 connection(s)
I20260926 22:51:17.832033 27257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
I20260926 22:51:17.832815 27325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:17.834374 27325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:17.841223 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 27257
I20260926 22:51:17.841313 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-1/wal/instance
I20260926 22:51:17.841519 13904 external_mini_cluster.cc:1428] Setting key 97afe9a9512e481916d60c57c5a6eb16
I20260926 22:51:17.842243 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.834983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.13.148.62:36763)
0926 22:51:17.835029 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:17.835030 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:17.835224 (+ 194us) server_negotiation.cc:407] Connection header received
0926 22:51:17.835678 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:17.835679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:17.835687 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:17.835733 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:17.836291 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.836746 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.837650 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.837865 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.837909 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.839778 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:17.839780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:17.839780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:17.839786 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:17.841015 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.841359 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.841360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.841361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.841424 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.841646 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.841647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.841647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.841791 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:17.841850 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:17.842064 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 22:51:17.842140 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:51:17.842311 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.834961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36763 (local address 127.0.0.1:49204)
0926 22:51:17.835123 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.835160 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:17.835578 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:17.835794 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:17.835797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:17.836043 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:17.836232 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.836236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.836806 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.836806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.837591 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.837594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.837926 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.837926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.838029 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.838449 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:17.838457 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:17.839688 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:17.841102 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.841105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.841106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.841301 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.841481 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.841482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.841482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.841596 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.841842 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:17.841846 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:17.841953 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 22:51:17.842032 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 22:51:17.842144 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260926 22:51:17.842804 27325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.60@KRBTEST.COM; defaulting to no policy
I20260926 22:51:17.849599 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.843030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:49560)
0926 22:51:17.843191 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.843201 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:17.843249 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:17.843371 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:17.843371 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:17.843492 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:17.843531 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.843533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.844573 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.844574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.845043 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.845044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.845630 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.845631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.845682 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.846024 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:17.846028 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:17.847025 (+ 997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:17.848708 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.848709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.848709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.848850 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.849035 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.849036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.849036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.849082 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.849338 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:17.849339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:17.849373 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 22:51:17.849425 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:51:17.849513 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 22:51:17.849715 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.843079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.13.148.61:37933)
0926 22:51:17.843207 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:17.843211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:17.843248 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:51:17.843298 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:17.843301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:17.843312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:17.843371 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:17.843589 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.844462 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.845117 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.845565 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.845634 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.847096 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:17.847104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:17.847104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:17.847110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:17.848627 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.848918 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.848921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.848921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.848980 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.849148 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.849149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.849150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.849281 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:17.849350 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:17.849460 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 22:51:17.849606 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260926 22:51:17.850201 27325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:51:17.850471 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.850379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49882)
0926 22:51:17.850415 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.850453 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 22:51:17.850580 27276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
I20260926 22:51:17.850927 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.850869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49896)
0926 22:51:17.850890 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.850911 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260926 22:51:17.851020 27325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42423: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
I20260926 22:51:17.863570 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:42423
--webserver_interface=127.13.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36763,127.13.148.61:37933,127.13.148.60:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:17.900228 27325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } attempt: 1
I20260926 22:51:17.900720 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.900589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49910)
0926 22:51:17.900650 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.900690 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 22:51:17.901093 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.900985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49918)
0926 22:51:17.901028 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.901060 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 22:51:17.901208 27325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42423: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
I20260926 22:51:17.909936 27250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } attempt: 3
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:51:17.916774 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.910294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:49566)
0926 22:51:17.910352 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.910363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:17.910420 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:17.910557 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:17.910558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:17.910705 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:17.910772 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.910774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.911696 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.911697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.912227 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.912229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.912713 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.912713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.912768 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.913171 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:17.913176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:17.914369 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:17.915929 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.915929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.915930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.916096 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.916265 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.916266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.916266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.916310 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.916526 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:17.916527 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:17.916563 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 22:51:17.916606 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:51:17.916696 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:51:17.916795 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.910335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.13.148.61:37933)
0926 22:51:17.910392 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:17.910394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:17.910408 (+ 14us) server_negotiation.cc:407] Connection header received
0926 22:51:17.910475 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:17.910476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:17.910488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:17.910541 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:17.910827 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.911596 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.912304 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.912625 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.912709 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.914441 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:17.914442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:17.914443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:17.914450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:17.915859 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.916151 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.916152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.916153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.916212 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.916354 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.916355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.916355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.916476 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:17.916529 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:17.916628 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 22:51:17.916710 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:51:17.917202 27250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:51:17.917495 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.917405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49930)
0926 22:51:17.917445 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.917470 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 22:51:17.917801 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.917703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49934)
0926 22:51:17.917761 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.917784 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 22:51:17.917876 27250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42423: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
W20260926 22:51:17.943588 27332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:17.943778 27332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:17.943805 27332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:17.944914 27332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:17.944955 27332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:17.944973 27332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:17.944988 27332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:17.945003 27332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:17.946511 27332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36763,127.13.148.61:37933,127.13.148.60:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:17.946743 27332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:17.946939 27332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:51:17.948998 27250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } attempt: 1
I20260926 22:51:17.949355 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.949249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49946)
0926 22:51:17.949299 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.949328 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 22:51:17.949698 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.949577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49958)
0926 22:51:17.949628 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.949669 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 22:51:17.949811 27250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42423: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
W20260926 22:51:17.951349 27337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:17.951337 27340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:17.951368 27338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:17.951573 27332 server_base.cc:1034] running on GCE node
I20260926 22:51:17.951773 27332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:17.951990 27332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:17.953138 27332 hybrid_clock.cc:648] HybridClock initialized: now 1790463077953125 us; error 29 us; skew 500 ppm
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:17.955367 27332 init.cc:377] Logged in from keytab as kudu/127.13.148.60@KRBTEST.COM (short username kudu)
I20260926 22:51:17.955760 27332 webserver.cc:460] Webserver started at http://127.13.148.60:43187/ using document root <none> and password file <none>
I20260926 22:51:17.955955 27332 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:17.956015 27332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:17.956135 27332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:17.957072 27332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/data/instance:
uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "4133f57c4512fb91f6db2ccf771234aa"
server_key_iv: "32de648b1fff820d63d4bf5e728d7fc3"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.957382 27332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/wal/instance:
uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "4133f57c4512fb91f6db2ccf771234aa"
server_key_iv: "32de648b1fff820d63d4bf5e728d7fc3"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.958688 27332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:51:17.959292 27347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:17.959431 27332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:51:17.959506 27332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/wal
uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc"
format_stamp: "Formatted at 2026-09-26 22:51:17 on dist-test-slave-98hb"
server_key: "4133f57c4512fb91f6db2ccf771234aa"
server_key_iv: "32de648b1fff820d63d4bf5e728d7fc3"
server_key_version: "encryptionkey@0"
I20260926 22:51:17.959559 27332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:17.969362 27325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } attempt: 2
I20260926 22:51:17.969769 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.969612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49962)
0926 22:51:17.969673 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.969713 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 22:51:17.970111 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.970015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49966)
0926 22:51:17.970057 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.970086 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 22:51:17.970196 27325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:42423: Network error: Client connection negotiation failed: client connection to 127.13.148.60:42423: connect: Connection refused (error 111)
I20260926 22:51:17.984163 27332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:17.985877 27332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:17.985987 27332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:17.989615 27332 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:42423
I20260926 22:51:17.989674 27399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:42423 every 8 connection(s)
I20260926 22:51:17.989990 27332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
I20260926 22:51:17.990844 27400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:17.992331 27400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:51:17 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:17.997658 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 27332
I20260926 22:51:17.997793 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790462935413609-13904-0/minicluster-data/master-2/wal/instance
I20260926 22:51:17.997963 13904 external_mini_cluster.cc:1428] Setting key 6b19df566f38d1bbdcf106e55d381e80
I20260926 22:51:18.000383 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.992966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.13.148.62:36763)
0926 22:51:17.993027 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:17.993029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:17.993211 (+ 182us) server_negotiation.cc:407] Connection header received
0926 22:51:17.993634 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:17.993635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:17.993643 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:17.993703 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:17.994289 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.994764 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.995674 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.995857 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.995909 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.997813 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:17.997815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:17.997815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:17.997823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:17.999257 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.999616 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.999617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.999617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.999677 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:17.999863 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:17.999864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:17.999864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:17.999985 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.000042 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.000255 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.000320 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 22:51:18.000610 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.992948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36763 (local address 127.0.0.1:49214)
0926 22:51:17.993092 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:17.993133 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:17.993566 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:17.993796 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:17.993800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:17.994031 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:17.994228 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.994232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.994822 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.994823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.995592 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:17.995594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:17.995914 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:17.995915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:17.996022 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:17.996483 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:17.996491 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:17.997712 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:17.999336 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.999341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.999343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.999545 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:17.999733 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:17.999733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:17.999734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:17.999817 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.000034 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:18.000037 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:18.000143 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:51:18.000233 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 22:51:18.000381 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260926 22:51:18.001097 27400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:51:18 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
Sep 26 22:51:18 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:51:18.006469 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:17.998298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.13.148.60:42423)
0926 22:51:17.998467 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:17.998472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:17.998526 (+ 54us) server_negotiation.cc:407] Connection header received
0926 22:51:17.998580 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:17.998582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:17.998593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:17.998649 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:17.999260 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.000138 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.001081 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.001477 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.001534 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.003836 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.003842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.003842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.003848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.005391 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.005778 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.005780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.005781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.005831 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.006009 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.006009 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.006009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.006122 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.006245 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.006312 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.006385 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260926 22:51:18.008234 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.001326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:37933 (local address 127.0.0.1:49570)
0926 22:51:18.001368 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:18.001378 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:18.001417 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:18.001520 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:18.001520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:18.001654 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:18.001703 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:18.001704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.002779 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.002780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:18.003234 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:18.003235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.003642 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.003643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:18.003701 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.004015 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:18.004020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:18.005315 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:18.007408 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:18.007408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:18.007409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:18.007606 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.007773 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:18.007774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:18.007774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:18.007826 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.008045 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:18.008045 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:18.008076 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 22:51:18.008118 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 22:51:18.008178 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:51:18.008308 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.001389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.13.148.61:37933)
0926 22:51:18.001430 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.001431 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.001439 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:51:18.001463 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.001464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.001473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.001516 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.001999 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.002710 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.003308 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.003572 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.003643 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.005391 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.005393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.005393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.005400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.007341 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.007668 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.007668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.007669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.007712 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.007872 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.007873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.007873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.007992 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.008052 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.008141 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.008222 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:51:18.008594 27400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:51:18 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
Sep 26 22:51:18 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:51:18.014653 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.008805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:49984)
0926 22:51:18.008842 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:18.008850 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:18.008880 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:18.008972 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:18.008972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:18.009092 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:18.009115 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:18.009116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.009765 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.009766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:18.010486 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:18.010488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.010853 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.010854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:18.010892 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.011211 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:18.011214 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:18.012429 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:18.013717 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:18.013718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:18.013718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:18.013913 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.014098 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:18.014099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:18.014099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:18.014157 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.014391 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:18.014392 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:18.014432 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 22:51:18.014495 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 22:51:18.014567 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 22:51:18.014686 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.007127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.13.148.62:36763)
0926 22:51:18.007163 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.007165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.007172 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:18.007191 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.007192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.007201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.007264 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.007745 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.008380 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.009132 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.009323 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.009371 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.011684 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.011686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.011686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.011694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.013613 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.013980 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.013980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.013981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.014036 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.014219 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.014220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.014220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.014331 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.014490 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.014527 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.014604 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:51:18.014782 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.008859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.13.148.60:42423)
0926 22:51:18.008886 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.008888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.008896 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:51:18.008913 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.008913 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.008922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.008971 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.009172 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.009667 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.010567 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.010781 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.010838 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.012488 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.012489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.012490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.012496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.013642 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.013979 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.013980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.013980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.014041 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.014214 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.014215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.014215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.014328 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.014384 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.014527 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.014615 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":15}
I20260926 22:51:18.016217 27400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: Bootstrap starting.
I20260926 22:51:18.017093 27400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:18.017387 27400 log.cc:824] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:18.018297 27400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: No bootstrap required, opened a new log
Sep 26 22:51:18 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:51:18.020007 27400 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.020157 27400 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:18.020201 27400 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff126c0b688a4de6ae9a9bb7adbbcadc, State: Initialized, Role: FOLLOWER
I20260926 22:51:18.020323 27400 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.020620 27400 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:18.020622 27406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.020776 27406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:51:18.021605 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.015286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.13.148.61:37933)
0926 22:51:18.015322 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.015324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.015331 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:18.015351 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.015352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.015360 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.015418 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.015791 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.016432 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.017016 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.017279 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.017351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.019094 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.019095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.019095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.019102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.020463 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.020814 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.020815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.020815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.020870 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.021044 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.021045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.021045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.021190 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.021405 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.021445 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.021520 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:51:18.022503 27417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:51:18.022559 27417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:51:18.022863 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:51:18.024945 27250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } attempt: 2
I20260926 22:51:18.022835 27332 master_runner.cc:186] Error getting master registration for 127.13.148.62:36763: OK, instance_id { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" instance_seqno: 1790463077681659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:51:18.025420 27332 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:51:18.031955 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.024361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.13.148.61:37933)
0926 22:51:18.024396 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.024397 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.024849 (+ 452us) server_negotiation.cc:407] Connection header received
0926 22:51:18.024934 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.024935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.024944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.024969 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.025799 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.026472 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.027443 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.027728 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.027766 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.029250 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.029251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.029252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.029258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.030774 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.031139 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.031141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.031141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.031192 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.031381 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.031382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.031382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.031497 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.031594 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.031792 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.031865 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:51:18.032951 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.024054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.13.148.62:36763)
0926 22:51:18.024094 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.024096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.024348 (+ 252us) server_negotiation.cc:407] Connection header received
0926 22:51:18.024435 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.024436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.024446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.024472 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.025613 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.026316 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.027273 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.027553 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.027590 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.029268 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.029270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.029270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.029276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.031035 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.031438 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.031439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.031439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.031494 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.031680 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.031681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.031682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.031793 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.031853 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.032241 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.032892 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":588,"server-negotiator.queue_time_us":20}
I20260926 22:51:18.034042 27404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.024612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.13.148.60:42423)
0926 22:51:18.024645 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.024646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.025668 (+ 1022us) server_negotiation.cc:407] Connection header received
0926 22:51:18.025796 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.025797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.025807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.025833 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.026458 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.027129 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.027976 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.028167 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.028218 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.029335 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.029336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.029337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.029342 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.031035 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.033199 (+ 2164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.033201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.033201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.033262 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.033471 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.033472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.033473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.033630 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.033724 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.033873 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.033952 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 22:51:18 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:51:18.037568 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.025284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.13.148.60:42423)
0926 22:51:18.029816 (+ 4532us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.029818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.029825 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:18.029842 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.029843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.029874 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.030218 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.031245 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.031981 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.033229 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.033516 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.033575 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.035400 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.035401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.035401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.035407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.036642 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.036976 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.036977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.036977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.037020 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.037175 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.037175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.037175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.037254 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.037294 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.037424 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.037476 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4513,"thread_start_us":81,"threads_started":1}
I20260926 22:51:18.037597 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.025166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:50000)
0926 22:51:18.025490 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:18.025501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:18.025540 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:18.030243 (+ 4703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:18.030244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:18.030363 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:18.030397 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:18.030398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.032185 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.032186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:18.032797 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:18.032799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.033592 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.033593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:18.033652 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.034160 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:18.034164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:18.035329 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:18.036714 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:18.036715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:18.036716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:18.036913 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.037058 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:18.037059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:18.037059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:18.037116 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.037301 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:18.037302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:18.037342 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 22:51:18.037411 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 22:51:18.037495 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260926 22:51:18.039057 27250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f: Bootstrap starting.
I20260926 22:51:18.039775 27250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:18.040060 27250 log.cc:824] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:18.040804 27250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f: No bootstrap required, opened a new log
I20260926 22:51:18.042771 27250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.042905 27250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:18.042944 27250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beb4a00aca043819746f9af5bdd208f, State: Initialized, Role: FOLLOWER
I20260926 22:51:18.043061 27250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.043311 27428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.043361 27250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:18.043387 27428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:51:18.045104 27439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:51:18.045154 27439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:51:18.045830 27182 master_runner.cc:186] Error getting master registration for 127.13.148.61:37933: OK, instance_id { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" instance_seqno: 1790463077830664 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:51:18.047825 27182 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:51:18.061327 27325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } attempt: 3
Sep 26 22:51:18 dist-test-slave-98hb krb5kdc[27169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463077, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:51:18.067564 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.061695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.13.148.60:42423)
0926 22:51:18.061751 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:18.061753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:18.061760 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:18.061818 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:18.061819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:18.061831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:18.061875 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:18.062149 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.062681 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.063494 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.063697 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.063752 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.065379 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:18.065380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:18.065381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:18.065387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:18.066623 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.066910 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.066911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.066911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.066964 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:18.067116 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:18.067117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:18.067117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:18.067224 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:18.067276 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:18.067403 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 22:51:18.067475 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:51:18.067564 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:18.061648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:42423 (local address 127.0.0.1:50008)
0926 22:51:18.061706 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:18.061716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:18.061771 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:18.061883 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:18.061884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:18.062041 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:18.062083 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:18.062085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.062748 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.062749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:18.063421 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:18.063423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:18.063754 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:18.063755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:18.063804 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:18.064194 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:18.064199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:18.065306 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:18.066695 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:18.066696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:18.066696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:18.066853 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.067013 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:18.067014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:18.067014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:18.067069 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:18.067275 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:18.067276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:18.067319 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:51:18.067378 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:51:18.067468 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 22:51:18.069046 27325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: Bootstrap starting.
I20260926 22:51:18.069669 27325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:18.069984 27325 log.cc:824] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:18.070725 27325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: No bootstrap required, opened a new log
I20260926 22:51:18.072608 27325 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.072719 27325 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:18.072751 27325 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b009e1df7441bc85e1b3eb7dbd63d3, State: Initialized, Role: FOLLOWER
I20260926 22:51:18.072870 27325 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.073110 27442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.073185 27325 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:18.073236 27442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 22:51:18.074833 27453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:51:18.074936 27453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 22:51:18.078583 27278 tablet.cc:2369] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:51:18.088927 27257 master_runner.cc:263] No leader master found from master d6b009e1df7441bc85e1b3eb7dbd63d3
I20260926 22:51:18.088979 27257 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 22:51:18.179806 27203 tablet.cc:2369] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:51:18.236553 27353 tablet.cc:2369] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:51:18.281610 27428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:51:18.281762 27428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.282014 27428 leader_election.cc:288] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b009e1df7441bc85e1b3eb7dbd63d3 (127.13.148.61:37933), ff126c0b688a4de6ae9a9bb7adbbcadc (127.13.148.60:42423)
I20260926 22:51:18.282305 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5beb4a00aca043819746f9af5bdd208f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" is_pre_election: true
I20260926 22:51:18.282307 27374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5beb4a00aca043819746f9af5bdd208f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" is_pre_election: true
I20260926 22:51:18.282447 27299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beb4a00aca043819746f9af5bdd208f in term 0.
I20260926 22:51:18.282469 27374 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beb4a00aca043819746f9af5bdd208f in term 0.
I20260926 22:51:18.282629 27201 leader_election.cc:302] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beb4a00aca043819746f9af5bdd208f, d6b009e1df7441bc85e1b3eb7dbd63d3; no voters:
I20260926 22:51:18.282755 27428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:51:18.282814 27428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:51:18.282836 27428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:18.283543 27428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.283649 27428 leader_election.cc:288] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [CANDIDATE]: Term 1 election: Requested vote from peers d6b009e1df7441bc85e1b3eb7dbd63d3 (127.13.148.61:37933), ff126c0b688a4de6ae9a9bb7adbbcadc (127.13.148.60:42423)
I20260926 22:51:18.283795 27299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5beb4a00aca043819746f9af5bdd208f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3"
I20260926 22:51:18.283834 27374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5beb4a00aca043819746f9af5bdd208f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc"
I20260926 22:51:18.283883 27299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:18.283905 27374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:18.284734 27299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beb4a00aca043819746f9af5bdd208f in term 1.
I20260926 22:51:18.284734 27374 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beb4a00aca043819746f9af5bdd208f in term 1.
I20260926 22:51:18.284901 27201 leader_election.cc:302] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beb4a00aca043819746f9af5bdd208f, ff126c0b688a4de6ae9a9bb7adbbcadc; no voters:
I20260926 22:51:18.285001 27428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:18.285140 27428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [term 1 LEADER]: Becoming Leader. State: Replica: 5beb4a00aca043819746f9af5bdd208f, State: Running, Role: LEADER
I20260926 22:51:18.285238 27428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } }
I20260926 22:51:18.285619 27455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5beb4a00aca043819746f9af5bdd208f. Latest consensus state: current_term: 1 leader_uuid: "5beb4a00aca043819746f9af5bdd208f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.285725 27455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:18.285952 27457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:51:18.286113 27457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:51:18.287321 27374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [term 1 FOLLOWER]: Refusing update from remote peer 5beb4a00aca043819746f9af5bdd208f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:18.287344 27299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [term 1 FOLLOWER]: Refusing update from remote peer 5beb4a00aca043819746f9af5bdd208f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:18.287603 27455 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:18.287710 27428 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:18.288820 27406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5beb4a00aca043819746f9af5bdd208f. Latest consensus state: current_term: 1 leader_uuid: "5beb4a00aca043819746f9af5bdd208f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.288889 27406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:51:18.288959 27442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5beb4a00aca043819746f9af5bdd208f. Latest consensus state: current_term: 1 leader_uuid: "5beb4a00aca043819746f9af5bdd208f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.289034 27442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:51:18.289901 27455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5beb4a00aca043819746f9af5bdd208f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.289983 27455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:18.290028 27458 mvcc.cc:205] Tried to move back new op lower bound from 7333736768662962176 to 7333736768656867328. Current Snapshot: MvccSnapshot[applied={T|T < 7333736768662962176}]
I20260926 22:51:18.290138 27442 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5beb4a00aca043819746f9af5bdd208f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.290175 27455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5beb4a00aca043819746f9af5bdd208f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.290221 27455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:18.290222 27442 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:51:18.290426 27460 mvcc.cc:205] Tried to move back new op lower bound from 7333736768662962176 to 7333736768656867328. Current Snapshot: MvccSnapshot[applied={T|T < 7333736768662962176 or (T in {7333736768662962176})}]
I20260926 22:51:18.290591 27406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5beb4a00aca043819746f9af5bdd208f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5beb4a00aca043819746f9af5bdd208f" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36763 } } peers { permanent_uuid: "d6b009e1df7441bc85e1b3eb7dbd63d3" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 37933 } } peers { permanent_uuid: "ff126c0b688a4de6ae9a9bb7adbbcadc" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 42423 } } }
I20260926 22:51:18.290678 27406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:51:18.290912 27457 catalog_manager.cc:1347] Generated new cluster ID: ab278f37b7a94a1c8a73011d8b8dc966
I20260926 22:51:18.290948 27457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:51:18.295678 27457 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:51:18.296226 27457 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:51:18.300835 27457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5beb4a00aca043819746f9af5bdd208f: Generated new TSK 0
I20260926 22:51:18.300992 27457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:51:19.022940 27417 catalog_manager.cc:1259] Loaded cluster ID: ab278f37b7a94a1c8a73011d8b8dc966
I20260926 22:51:19.023020 27417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: loading cluster ID for follower catalog manager: success
I20260926 22:51:19.023844 27417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: acquiring CA information for follower catalog manager: success
I20260926 22:51:19.024140 27417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff126c0b688a4de6ae9a9bb7adbbcadc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:51:19.075382 27453 catalog_manager.cc:1259] Loaded cluster ID: ab278f37b7a94a1c8a73011d8b8dc966
I20260926 22:51:19.075456 27453 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: loading cluster ID for follower catalog manager: success
I20260926 22:51:19.076267 27453 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: acquiring CA information for follower catalog manager: success
I20260926 22:51:19.076562 27453 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6b009e1df7441bc85e1b3eb7dbd63d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:51:19.354427 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 27182
I20260926 22:51:19.359110 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 27257
I20260926 22:51:19.363269 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 27332
2026-09-26T22:51:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2845 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:51:19 dist-test-slave-98hb krb5kdc[27473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:51:19 dist-test-slave-98hb krb5kdc[27473](info): set up 2 sockets
Sep 26 22:51:19 dist-test-slave-98hb krb5kdc[27473](info): commencing operation
krb5kdc: starting...
W20260926 22:51:20.301817 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:51:20 dist-test-slave-98hb krb5kdc[27473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:51:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:51:20.406394 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44163
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:35281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:20.483451 27486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:20.483632 27486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:20.483652 27486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:20.484743 27486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:20.484782 27486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:20.484795 27486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:20.484807 27486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:20.484817 27486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:20.486254 27486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:20.486457 27486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:20.486653 27486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:20.490994 27494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:20.491006 27491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:20.491133 27486 server_base.cc:1034] running on GCE node
W20260926 22:51:20.491005 27492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:20.491416 27486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:20.491701 27486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:20.492842 27486 hybrid_clock.cc:648] HybridClock initialized: now 1790463080492823 us; error 32 us; skew 500 ppm
Sep 26 22:51:20 dist-test-slave-98hb krb5kdc[27473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463080, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:20.495095 27486 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:51:20.495486 27486 webserver.cc:460] Webserver started at http://127.13.148.62:35567/ using document root <none> and password file <none>
I20260926 22:51:20.495685 27486 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:20.495728 27486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:20.495846 27486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:20.496832 27486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "1e34d938f9514223b071dcf5b98d596e"
format_stamp: "Formatted at 2026-09-26 22:51:20 on dist-test-slave-98hb"
server_key: "a1cbadad5c9e5fe8cfec3a61ae0134db"
server_key_iv: "b2ca055c97862dab18296e080e4a6db9"
server_key_version: "encryptionkey@0"
I20260926 22:51:20.497141 27486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "1e34d938f9514223b071dcf5b98d596e"
format_stamp: "Formatted at 2026-09-26 22:51:20 on dist-test-slave-98hb"
server_key: "a1cbadad5c9e5fe8cfec3a61ae0134db"
server_key_iv: "b2ca055c97862dab18296e080e4a6db9"
server_key_version: "encryptionkey@0"
I20260926 22:51:20.498348 27486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:51:20.498936 27501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:20.499084 27486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:51:20.499172 27486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "1e34d938f9514223b071dcf5b98d596e"
format_stamp: "Formatted at 2026-09-26 22:51:20 on dist-test-slave-98hb"
server_key: "a1cbadad5c9e5fe8cfec3a61ae0134db"
server_key_iv: "b2ca055c97862dab18296e080e4a6db9"
server_key_version: "encryptionkey@0"
I20260926 22:51:20.499224 27486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:20.509249 27486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:20.511143 27486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:20.511253 27486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:20.514722 27486 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:44163
I20260926 22:51:20.514752 27553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:44163 every 8 connection(s)
I20260926 22:51:20.515033 27486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:51:20.515801 27554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:20.517931 27554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e: Bootstrap starting.
I20260926 22:51:20.518436 27554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:20.518692 27554 log.cc:824] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:20.519284 27554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e: No bootstrap required, opened a new log
I20260926 22:51:20.520359 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 27486
I20260926 22:51:20.520437 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:51:20.520601 13904 external_mini_cluster.cc:1428] Setting key 8be1878776b475c2e5c6104b842b1ef1
I20260926 22:51:20.520955 27554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e34d938f9514223b071dcf5b98d596e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44163 } }
I20260926 22:51:20.521186 27554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:20.521242 27554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e34d938f9514223b071dcf5b98d596e, State: Initialized, Role: FOLLOWER
I20260926 22:51:20.521359 27554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e34d938f9514223b071dcf5b98d596e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44163 } }
I20260926 22:51:20.521435 27554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:51:20.521469 27554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:51:20.521513 27554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:20.522439 27554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e34d938f9514223b071dcf5b98d596e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44163 } }
I20260926 22:51:20.522540 27554 leader_election.cc:302] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e34d938f9514223b071dcf5b98d596e; no voters:
I20260926 22:51:20.522692 27554 leader_election.cc:288] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:51:20.522728 27559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:20.522853 27559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [term 1 LEADER]: Becoming Leader. State: Replica: 1e34d938f9514223b071dcf5b98d596e, State: Running, Role: LEADER
I20260926 22:51:20.522917 27554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:20.522972 27559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e34d938f9514223b071dcf5b98d596e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44163 } }
I20260926 22:51:20.523277 27559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e34d938f9514223b071dcf5b98d596e. Latest consensus state: current_term: 1 leader_uuid: "1e34d938f9514223b071dcf5b98d596e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e34d938f9514223b071dcf5b98d596e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44163 } } }
I20260926 22:51:20.523470 27559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:20.523411 27561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e34d938f9514223b071dcf5b98d596e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e34d938f9514223b071dcf5b98d596e" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 44163 } } }
I20260926 22:51:20.523829 27561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e [sys.catalog]: This master's current role is: LEADER
W20260926 22:51:20.524467 27575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:51:20.524513 27575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:51:20.524557 27568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:51:20.524708 27568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:51:20.526074 27568 catalog_manager.cc:1347] Generated new cluster ID: 6f89135f29d144cdacf8fbaded9ca5b1
I20260926 22:51:20.526127 27568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:51:20 dist-test-slave-98hb krb5kdc[27473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:20.530422 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:20.521019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.13.148.62:44163)
0926 22:51:20.521207 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:20.521216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:20.521248 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:51:20.521357 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:20.521363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:20.521416 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:20.521564 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:20.522165 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:20.523513 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:20.524567 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:20.525125 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:20.525237 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:20.527408 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:20.527412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:20.527412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:20.527423 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:20.528838 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:20.529131 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:20.529134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:20.529135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:20.529222 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:20.529388 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:20.529388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:20.529389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:20.529602 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:20.530135 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:20.530196 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 22:51:20.530291 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260926 22:51:20.531291 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:51:20.542768 27568 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:51:20.543241 27568 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:51:20.545794 27568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e34d938f9514223b071dcf5b98d596e: Generated new TSK 0
I20260926 22:51:20.545928 27568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:51:20.584733 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 27486
2026-09-26T22:51:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1220 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:51:20 dist-test-slave-98hb krb5kdc[27581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:51:20 dist-test-slave-98hb krb5kdc[27581](info): set up 2 sockets
Sep 26 22:51:20 dist-test-slave-98hb krb5kdc[27581](info): commencing operation
krb5kdc: starting...
W20260926 22:51:21.525885 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:51:21 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:51:21.632148 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39089
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:21.709316 27594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:21.709511 27594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:21.709542 27594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:21.710692 27594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:21.710734 27594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:21.710752 27594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:21.710768 27594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:21.710783 27594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:21.712210 27594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:21.712447 27594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:21.712636 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:51:21.717170 27594 server_base.cc:1034] running on GCE node
W20260926 22:51:21.717095 27600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:21.717101 27602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:21.717108 27599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:21.717499 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:21.717808 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:21.718946 27594 hybrid_clock.cc:648] HybridClock initialized: now 1790463081718932 us; error 34 us; skew 500 ppm
Sep 26 22:51:21 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:21.721167 27594 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:51:21.721535 27594 webserver.cc:460] Webserver started at http://127.13.148.62:35931/ using document root <none> and password file <none>
I20260926 22:51:21.721764 27594 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:21.721817 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:21.721935 27594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:21.722904 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "ca576ae926344e978d3cb1f2bec9f6a6"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "50c8fd20756112ed9e8e0ed44f2bb71f"
server_key_iv: "6f17e269e03ebb94e918b626460855fa"
server_key_version: "encryptionkey@0"
I20260926 22:51:21.723218 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "ca576ae926344e978d3cb1f2bec9f6a6"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "50c8fd20756112ed9e8e0ed44f2bb71f"
server_key_iv: "6f17e269e03ebb94e918b626460855fa"
server_key_version: "encryptionkey@0"
I20260926 22:51:21.724359 27594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:51:21.724944 27609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:21.725087 27594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:51:21.725164 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "ca576ae926344e978d3cb1f2bec9f6a6"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "50c8fd20756112ed9e8e0ed44f2bb71f"
server_key_iv: "6f17e269e03ebb94e918b626460855fa"
server_key_version: "encryptionkey@0"
I20260926 22:51:21.725227 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:21.763012 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:21.765084 27594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:21.765218 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:21.768926 27594 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39089
I20260926 22:51:21.768971 27661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39089 every 8 connection(s)
I20260926 22:51:21.769371 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:51:21.770397 27662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:21.772601 27662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Bootstrap starting.
I20260926 22:51:21.773129 27662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:21.773380 27662 log.cc:824] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:21.774006 27662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: No bootstrap required, opened a new log
I20260926 22:51:21.775537 27662 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:21.775668 27662 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:21.775691 27662 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca576ae926344e978d3cb1f2bec9f6a6, State: Initialized, Role: FOLLOWER
I20260926 22:51:21.775821 27662 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:21.775879 27662 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:51:21.775899 27662 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:51:21.775942 27662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:21.776361 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 27594
I20260926 22:51:21.776458 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:51:21.776638 13904 external_mini_cluster.cc:1428] Setting key 7ae2d70a5f4b38c7b4a424fe65019d35
I20260926 22:51:21.776762 27662 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:21.776849 27662 leader_election.cc:302] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca576ae926344e978d3cb1f2bec9f6a6; no voters:
I20260926 22:51:21.776991 27662 leader_election.cc:288] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:51:21.777213 27667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:21.777240 27662 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:21.777402 27667 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 1 LEADER]: Becoming Leader. State: Replica: ca576ae926344e978d3cb1f2bec9f6a6, State: Running, Role: LEADER
I20260926 22:51:21.777487 27667 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:21.777856 27667 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca576ae926344e978d3cb1f2bec9f6a6. Latest consensus state: current_term: 1 leader_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } } }
I20260926 22:51:21.777868 27669 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } } }
I20260926 22:51:21.777957 27667 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:21.777969 27669 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:21.778293 27676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:51:21.778825 27676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:51:21.780892 27676 catalog_manager.cc:1347] Generated new cluster ID: cb81e8c1abe24f1fba0a155be30314c9
I20260926 22:51:21.780944 27676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:51:21 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:21.788141 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:21.777044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.13.148.62:39089)
0926 22:51:21.777198 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:21.777207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:21.777241 (+ 34us) server_negotiation.cc:407] Connection header received
0926 22:51:21.777383 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:21.777390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:21.777455 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:21.778145 (+ 690us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:21.778413 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:21.779977 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:21.780842 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:21.781421 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:21.781517 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:21.784145 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:21.784150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:21.784150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:21.784162 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:21.786606 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:21.786969 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:21.786972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:21.786972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:21.787087 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:21.787257 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:21.787258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:21.787258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:21.787554 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:21.787730 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:21.787795 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 22:51:21.787931 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260926 22:51:21.794049 27676 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:51:21.794754 27676 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:51:21.802351 27676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Generated new TSK 0
I20260926 22:51:21.802512 27676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:51:21.832069 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39089
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:21.909021 27688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:21.909209 27688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:21.909241 27688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:21.910362 27688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:21.910400 27688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:21.910434 27688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:51:21.911819 27688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:21.912045 27688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:21.912246 27688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:21.916858 27693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:21.916920 27696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:21.916858 27694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:21.916970 27688 server_base.cc:1034] running on GCE node
I20260926 22:51:21.917271 27688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:21.917483 27688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:21.918613 27688 hybrid_clock.cc:648] HybridClock initialized: now 1790463081918598 us; error 32 us; skew 500 ppm
Sep 26 22:51:21 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:21.920904 27688 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:51:21.921317 27688 webserver.cc:460] Webserver started at http://127.13.148.1:42071/ using document root <none> and password file <none>
I20260926 22:51:21.921516 27688 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:21.921558 27688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:21.921684 27688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:21.922696 27688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "1e802fa25d224c5cb412a67835daacf0"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "a2604e20a0c6cbb07670486c60388bd0"
server_key_iv: "6d606a414ec88645c0ffe2f48cb4e450"
server_key_version: "encryptionkey@0"
I20260926 22:51:21.922998 27688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "1e802fa25d224c5cb412a67835daacf0"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "a2604e20a0c6cbb07670486c60388bd0"
server_key_iv: "6d606a414ec88645c0ffe2f48cb4e450"
server_key_version: "encryptionkey@0"
I20260926 22:51:21.924147 27688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:51:21.924782 27703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:21.924921 27688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:51:21.924978 27688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "1e802fa25d224c5cb412a67835daacf0"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "a2604e20a0c6cbb07670486c60388bd0"
server_key_iv: "6d606a414ec88645c0ffe2f48cb4e450"
server_key_version: "encryptionkey@0"
I20260926 22:51:21.925042 27688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:21.941465 27688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:21.943068 27688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:21.943183 27688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:21.943375 27688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:21.943645 27688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:21.943683 27688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:21.943703 27688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:21.943725 27688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:21.948876 27688 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:42817
I20260926 22:51:21.948941 27816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:42817 every 8 connection(s)
I20260926 22:51:21.949198 27688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:51:21 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463081, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:21.956436 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 27688
I20260926 22:51:21.956553 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:51:21.956760 13904 external_mini_cluster.cc:1428] Setting key 884a640a8aece19a5c5a62464a12a1fa
I20260926 22:51:21.957681 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:21.949817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:56155 (local address 127.13.148.62:39089)
0926 22:51:21.949860 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:21.949862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:21.950045 (+ 183us) server_negotiation.cc:407] Connection header received
0926 22:51:21.950495 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:21.950496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:21.950508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:21.950551 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:21.951369 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:21.951858 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:21.952733 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:21.952920 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:21.952963 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:21.954922 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:21.954925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:21.954925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:21.954933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:21.956296 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:21.956740 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:21.956740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:21.956741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:21.956792 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:21.957048 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:21.957048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:21.957048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:21.957153 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:21.957211 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:21.957507 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 22:51:21.957586 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:51:21.957845 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:21.949817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:56155)
0926 22:51:21.949950 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:21.949983 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:21.950422 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:21.950609 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:21.950613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:21.951116 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:21.951307 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:21.951312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:21.951916 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:21.951917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:21.952677 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:21.952678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:21.952975 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:21.952976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:21.953066 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:21.953495 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:21.953503 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:21.954817 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:21.956367 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:21.956375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:21.956377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:21.956668 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:21.956857 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:21.956858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:21.956858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:21.956982 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:21.957217 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:21.957224 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:21.957372 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 22:51:21.957477 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 22:51:21.957620 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260926 22:51:21.958962 27817 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39089
I20260926 22:51:21.959038 27817 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:21.959196 27817 heartbeater.cc:511] Master 127.13.148.62:39089 requested a full tablet report, sending...
I20260926 22:51:21.959628 27626 ts_manager.cc:194] Registered new tserver with Master: 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1:42817)
I20260926 22:51:21.960180 27626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:56155
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:51:21.979924 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39089
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:22.057502 27822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:22.057705 27822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:22.057726 27822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:22.058868 27822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:22.058908 27822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:22.058935 27822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:51:22.060344 27822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:22.060557 27822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:22.060767 27822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:22.065407 27828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.065408 27830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:22.065442 27822 server_base.cc:1034] running on GCE node
W20260926 22:51:22.065428 27827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:22.065819 27822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:22.066054 27822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:22.067194 27822 hybrid_clock.cc:648] HybridClock initialized: now 1790463082067185 us; error 28 us; skew 500 ppm
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:22.069666 27822 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:51:22.070175 27822 webserver.cc:460] Webserver started at http://127.13.148.2:40895/ using document root <none> and password file <none>
I20260926 22:51:22.070353 27822 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:22.070390 27822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:22.070480 27822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:22.071455 27822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "c26b5143e126414391cc065ab2288b2e"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "7ce0c5aca26bbc68702b952d777fb20a"
server_key_iv: "7ecb5bd2222fbd8cf74d51f9bba8633b"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.071749 27822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "c26b5143e126414391cc065ab2288b2e"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "7ce0c5aca26bbc68702b952d777fb20a"
server_key_iv: "7ecb5bd2222fbd8cf74d51f9bba8633b"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.072880 27822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:51:22.073505 27837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.073661 27822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:51:22.073764 27822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "c26b5143e126414391cc065ab2288b2e"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "7ce0c5aca26bbc68702b952d777fb20a"
server_key_iv: "7ecb5bd2222fbd8cf74d51f9bba8633b"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.073836 27822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:22.087013 27822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:22.088611 27822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:22.088719 27822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:22.088896 27822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:22.089197 27822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:22.089231 27822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.089251 27822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:22.089274 27822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.094359 27822 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:46319
I20260926 22:51:22.094416 27950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:46319 every 8 connection(s)
I20260926 22:51:22.094691 27822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:22.103451 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.095325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:35265 (local address 127.13.148.62:39089)
0926 22:51:22.095387 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.095388 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.095571 (+ 183us) server_negotiation.cc:407] Connection header received
0926 22:51:22.096051 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.096052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.096062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.096116 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:22.096988 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.097475 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.098455 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.098693 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.098747 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.100586 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:22.100588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:22.100588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:22.100594 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:22.102032 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.102458 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.102459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.102460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.102512 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.102773 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.102773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.102774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.102865 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:22.102908 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.103259 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.103359 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:51:22.103606 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.095292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:35265)
0926 22:51:22.095453 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.095494 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.095955 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.096177 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.096180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.096715 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:22.096900 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.096904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.097540 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.097541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.098379 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.098381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.098764 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.098765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.098870 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.099277 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:22.099285 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:22.100509 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:22.102105 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.102112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.102114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.102392 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.102567 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.102568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.102569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.102714 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.102917 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:22.102923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:22.103083 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.103228 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.103385 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260926 22:51:22.104039 27951 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39089
I20260926 22:51:22.104154 27951 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:22.104292 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 27822
I20260926 22:51:22.104337 27951 heartbeater.cc:511] Master 127.13.148.62:39089 requested a full tablet report, sending...
I20260926 22:51:22.104394 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:51:22.104558 13904 external_mini_cluster.cc:1428] Setting key 56caef86884196425a01bf075d559820
I20260926 22:51:22.104733 27626 ts_manager.cc:194] Registered new tserver with Master: c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319)
I20260926 22:51:22.105320 27626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:35265
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:51:22.125934 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39089
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:22.203306 27956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:22.203500 27956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:22.203522 27956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:22.204607 27956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:22.204641 27956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:22.204674 27956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:51:22.206053 27956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:22.206230 27956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:22.206395 27956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:51:22.211009 27956 server_base.cc:1034] running on GCE node
W20260926 22:51:22.210966 27962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.210978 27964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.210980 27961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:22.211329 27956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:22.211555 27956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:22.212697 27956 hybrid_clock.cc:648] HybridClock initialized: now 1790463082212681 us; error 29 us; skew 500 ppm
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:22.214929 27956 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:51:22.215313 27956 webserver.cc:460] Webserver started at http://127.13.148.3:33201/ using document root <none> and password file <none>
I20260926 22:51:22.215497 27956 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:22.215543 27956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:22.215669 27956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:22.216667 27956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "149fd66f51e743bc802e329050394a89"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "60cfe2b2031424ebdd78dc2addd3f0dc"
server_key_iv: "90b9fb5033d8a57e049a820a251334c2"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.216981 27956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "149fd66f51e743bc802e329050394a89"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "60cfe2b2031424ebdd78dc2addd3f0dc"
server_key_iv: "90b9fb5033d8a57e049a820a251334c2"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.218231 27956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:51:22.218854 27971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.218999 27956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:51:22.219079 27956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "149fd66f51e743bc802e329050394a89"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "60cfe2b2031424ebdd78dc2addd3f0dc"
server_key_iv: "90b9fb5033d8a57e049a820a251334c2"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.219136 27956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:22.225476 27956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:22.227078 27956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:22.227203 27956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:22.227420 27956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:22.227711 27956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:22.227784 27956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.227823 27956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:22.227836 27956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.233167 27956 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:34859
I20260926 22:51:22.233218 28084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:34859 every 8 connection(s)
I20260926 22:51:22.233489 27956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:22.239982 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 27956
I20260926 22:51:22.240094 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:51:22.240265 13904 external_mini_cluster.cc:1428] Setting key 4ae5c898293e0ec1f752f600f7f9daf6
I20260926 22:51:22.241649 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.234159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:33451 (local address 127.13.148.62:39089)
0926 22:51:22.234216 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.234218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.234389 (+ 171us) server_negotiation.cc:407] Connection header received
0926 22:51:22.234852 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.234853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.234862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.234909 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:22.235742 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.236235 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.237150 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.237344 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.237394 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.239262 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:22.239263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:22.239264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:22.239271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:22.240447 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.240897 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.240898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.240898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.240962 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.241194 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.241195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.241195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.241295 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:22.241341 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.241532 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.241588 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:51:22.241868 28087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.234140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:33451)
0926 22:51:22.234282 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.234317 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.234780 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.234962 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.234965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.235484 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:22.235679 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.235684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.236298 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.236299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.237073 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.237075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.237403 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.237404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.237496 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.237947 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:22.237954 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:22.239183 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:22.240538 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.240545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.240547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.240835 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.241020 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.241021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.241021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.241146 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.241333 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:22.241336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:22.241439 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.241512 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.241624 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260926 22:51:22.242266 28085 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39089
I20260926 22:51:22.242349 28085 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:22.242483 28085 heartbeater.cc:511] Master 127.13.148.62:39089 requested a full tablet report, sending...
I20260926 22:51:22.242797 27626 ts_manager.cc:194] Registered new tserver with Master: 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859)
I20260926 22:51:22.243110 27626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:33451
I20260926 22:51:22.251756 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:51:22.257920 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.252979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.13.148.62:39089)
0926 22:51:22.253031 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.253032 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.253119 (+ 87us) server_negotiation.cc:407] Connection header received
0926 22:51:22.253157 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.253157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.253171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.253212 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:22.253693 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.254185 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.254842 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.255012 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.255056 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.256049 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:22.256049 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:22.256050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:22.256054 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:22.257075 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.257344 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.257344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.257345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.257385 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.257494 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.257494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.257494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.257611 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:22.257677 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.257800 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.257858 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:51:22.259333 27626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:51:22.260825 27626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:51:22.266798 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.263016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46319 (local address 127.0.0.1:46182)
0926 22:51:22.263378 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.263386 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.263425 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.263643 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.263644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.263646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:22.263690 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.263692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.264529 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.264530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.265677 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.265679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.266511 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.266512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.266587 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.266602 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.266676 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.266725 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":63,"threads_started":1}
I20260926 22:51:22.266825 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.263128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.13.148.2:46319)
0926 22:51:22.263267 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.263271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.263426 (+ 155us) server_negotiation.cc:407] Connection header received
0926 22:51:22.263487 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.263490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.263505 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.263570 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:22.263759 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.264441 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.265779 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.266432 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.266505 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.266562 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.266707 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.266748 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260926 22:51:22.267082 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.262909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.13.148.1:42817)
0926 22:51:22.263167 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.263170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.263259 (+ 89us) server_negotiation.cc:407] Connection header received
0926 22:51:22.263360 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.263362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.263379 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.263459 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:22.263718 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.264831 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.265994 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.266687 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.266748 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.266799 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.266972 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.267015 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
I20260926 22:51:22.267060 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.262797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42817 (local address 127.0.0.1:49234)
0926 22:51:22.263139 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.263183 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.263271 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.263558 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.263562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.263564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:22.263634 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.263637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.264923 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.264924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.265923 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.265925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.266833 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.266834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.266898 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.266903 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.266955 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.266996 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260926 22:51:22.267716 27885 tablet_service.cc:1467] Processing CreateTablet for tablet e12abe023e0d42bdb0ce66b4adb50068 (DEFAULT_TABLE table=test-table [id=4044005d78f346f1b57679f66fffe110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:22.267832 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.262800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34859 (local address 127.0.0.1:41746)
0926 22:51:22.263284 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.263293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.263341 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.263573 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.263574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.263575 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:22.263634 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.263636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.265988 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.265989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.266789 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.266791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.267608 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.267609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.267677 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.267680 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.267724 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.267759 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260926 22:51:22.267835 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.262964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.13.148.3:34859)
0926 22:51:22.263140 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.263146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.263340 (+ 194us) server_negotiation.cc:407] Connection header received
0926 22:51:22.263402 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.263405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.263419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.263503 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:22.263718 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.264822 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.266874 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.267538 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.267596 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.267646 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.267734 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.267768 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
I20260926 22:51:22.268077 27885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12abe023e0d42bdb0ce66b4adb50068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:22.268415 28019 tablet_service.cc:1467] Processing CreateTablet for tablet e12abe023e0d42bdb0ce66b4adb50068 (DEFAULT_TABLE table=test-table [id=4044005d78f346f1b57679f66fffe110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:22.268694 28019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12abe023e0d42bdb0ce66b4adb50068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:22.270449 27751 tablet_service.cc:1467] Processing CreateTablet for tablet e12abe023e0d42bdb0ce66b4adb50068 (DEFAULT_TABLE table=test-table [id=4044005d78f346f1b57679f66fffe110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:22.270771 27751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12abe023e0d42bdb0ce66b4adb50068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:22.270893 28099 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Bootstrap starting.
I20260926 22:51:22.271469 28100 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: Bootstrap starting.
I20260926 22:51:22.271521 28099 tablet_bootstrap.cc:654] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:22.272097 28100 tablet_bootstrap.cc:654] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:22.272107 28099 log.cc:824] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:22.272576 28100 log.cc:824] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:22.273083 28099 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: No bootstrap required, opened a new log
I20260926 22:51:22.273144 28099 ts_tablet_manager.cc:1397] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:51:22.273278 28100 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: No bootstrap required, opened a new log
I20260926 22:51:22.273347 28100 ts_tablet_manager.cc:1397] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:22.273626 28103 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Bootstrap starting.
I20260926 22:51:22.274307 28103 tablet_bootstrap.cc:654] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:22.274883 28103 log.cc:824] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:22.274830 28100 raft_consensus.cc:348] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.274945 28100 raft_consensus.cc:374] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:22.274974 28100 raft_consensus.cc:729] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149fd66f51e743bc802e329050394a89, State: Initialized, Role: FOLLOWER
I20260926 22:51:22.275060 28100 consensus_queue.cc:260] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.275106 28099 raft_consensus.cc:348] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.275290 28099 raft_consensus.cc:374] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:22.275305 28100 ts_tablet_manager.cc:1428] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 22:51:22.275334 28099 raft_consensus.cc:729] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26b5143e126414391cc065ab2288b2e, State: Initialized, Role: FOLLOWER
I20260926 22:51:22.275305 28085 heartbeater.cc:503] Master 127.13.148.62:39089 was elected leader, sending a full tablet report...
I20260926 22:51:22.275467 28099 consensus_queue.cc:260] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.275741 28099 ts_tablet_manager.cc:1428] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:51:22.275763 27951 heartbeater.cc:503] Master 127.13.148.62:39089 was elected leader, sending a full tablet report...
I20260926 22:51:22.276440 28103 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: No bootstrap required, opened a new log
I20260926 22:51:22.276507 28103 ts_tablet_manager.cc:1397] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 22:51:22.278090 28103 raft_consensus.cc:348] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.278250 28103 raft_consensus.cc:374] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:22.278297 28103 raft_consensus.cc:729] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e802fa25d224c5cb412a67835daacf0, State: Initialized, Role: FOLLOWER
I20260926 22:51:22.278398 28103 consensus_queue.cc:260] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.278626 28103 ts_tablet_manager.cc:1428] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:51:22.278692 27817 heartbeater.cc:503] Master 127.13.148.62:39089 was elected leader, sending a full tablet report...
I20260926 22:51:22.333851 28106 raft_consensus.cc:482] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:51:22.333981 28106 raft_consensus.cc:504] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.334322 28106 leader_election.cc:288] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859), 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1:42817)
I20260926 22:51:22.337247 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.334500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42817 (local address 127.13.148.2:45659)
0926 22:51:22.334708 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.334716 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.334763 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.334869 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.334870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.334871 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:22.334942 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.334944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.335536 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.335536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.336481 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.336483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.337062 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.337063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.337113 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.337116 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.337153 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.337184 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":35,"thread_start_us":102,"threads_started":1}
I20260926 22:51:22.337277 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.334555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:45659 (local address 127.13.148.1:42817)
0926 22:51:22.334598 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.334600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.334772 (+ 172us) server_negotiation.cc:407] Connection header received
0926 22:51:22.334803 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.334804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.334813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.334860 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:22.335006 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.335478 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.336562 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.336990 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.337058 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.337066 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.337177 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.337211 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:51:22.337559 27771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12abe023e0d42bdb0ce66b4adb50068" candidate_uuid: "c26b5143e126414391cc065ab2288b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e802fa25d224c5cb412a67835daacf0" is_pre_election: true
I20260926 22:51:22.337714 27771 raft_consensus.cc:2454] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c26b5143e126414391cc065ab2288b2e in term 0.
I20260926 22:51:22.337875 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.334500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34859 (local address 127.13.148.2:60837)
0926 22:51:22.334579 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.334591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.334632 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.334905 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.334906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.334907 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:22.334948 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.334949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.336070 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.336071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.336941 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.336943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.337614 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.337615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.337693 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.337696 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.337765 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.337811 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 22:51:22.337873 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.334570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:60837 (local address 127.13.148.3:34859)
0926 22:51:22.334616 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.334619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.334627 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:51:22.334701 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.334702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.334712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.334768 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:22.335554 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.336000 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.337013 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.337537 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.337600 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.337607 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.337775 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.337810 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:51:22.337968 27839 leader_election.cc:302] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e802fa25d224c5cb412a67835daacf0, c26b5143e126414391cc065ab2288b2e; no voters:
I20260926 22:51:22.338155 28039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12abe023e0d42bdb0ce66b4adb50068" candidate_uuid: "c26b5143e126414391cc065ab2288b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149fd66f51e743bc802e329050394a89" is_pre_election: true
I20260926 22:51:22.338290 28039 raft_consensus.cc:2454] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c26b5143e126414391cc065ab2288b2e in term 0.
I20260926 22:51:22.338399 28106 raft_consensus.cc:2781] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:51:22.338461 28106 raft_consensus.cc:482] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:51:22.338485 28106 raft_consensus.cc:3037] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:22.339160 28106 raft_consensus.cc:504] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.339272 28106 leader_election.cc:288] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [CANDIDATE]: Term 1 election: Requested vote from peers 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859), 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1:42817)
I20260926 22:51:22.339427 27771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12abe023e0d42bdb0ce66b4adb50068" candidate_uuid: "c26b5143e126414391cc065ab2288b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e802fa25d224c5cb412a67835daacf0"
I20260926 22:51:22.339430 28039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12abe023e0d42bdb0ce66b4adb50068" candidate_uuid: "c26b5143e126414391cc065ab2288b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149fd66f51e743bc802e329050394a89"
I20260926 22:51:22.339500 27771 raft_consensus.cc:3037] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:22.339502 28039 raft_consensus.cc:3037] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:22.340179 28039 raft_consensus.cc:2454] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c26b5143e126414391cc065ab2288b2e in term 1.
I20260926 22:51:22.340188 27771 raft_consensus.cc:2454] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c26b5143e126414391cc065ab2288b2e in term 1.
I20260926 22:51:22.340346 27841 leader_election.cc:302] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149fd66f51e743bc802e329050394a89, c26b5143e126414391cc065ab2288b2e; no voters:
I20260926 22:51:22.340431 28106 raft_consensus.cc:2781] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:22.340561 28106 raft_consensus.cc:686] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 1 LEADER]: Becoming Leader. State: Replica: c26b5143e126414391cc065ab2288b2e, State: Running, Role: LEADER
I20260926 22:51:22.340668 28106 consensus_queue.cc:237] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.341428 27626 catalog_manager.cc:5495] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to c26b5143e126414391cc065ab2288b2e (127.13.148.2). New cstate: current_term: 1 leader_uuid: "c26b5143e126414391cc065ab2288b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } health_report { overall_health: HEALTHY } } }
W20260926 22:51:22.345136 27952 tablet.cc:2369] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:51:22.358824 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.356828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.13.148.2:46319)
0926 22:51:22.356875 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.356876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.356899 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:51:22.356967 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.356968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.356983 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.357037 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:51:22.357127 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.357577 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.358282 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.358457 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.358499 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.358583 (+ 84us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:51:22.358673 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:51:22.358724 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.358736 (+ 12us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.358750 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:51:22.360381 28039 raft_consensus.cc:1261] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Refusing update from remote peer c26b5143e126414391cc065ab2288b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:22.360383 27771 raft_consensus.cc:1261] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Refusing update from remote peer c26b5143e126414391cc065ab2288b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:22.360697 28109 consensus_queue.cc:1035] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:22.360812 28106 consensus_queue.cc:1035] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:22.363036 28112 mvcc.cc:205] Tried to move back new op lower bound from 7333736785345851392 to 7333736785267970048. Current Snapshot: MvccSnapshot[applied={T|T < 7333736785345851392}]
I20260926 22:51:22.363826 28113 mvcc.cc:205] Tried to move back new op lower bound from 7333736785345851392 to 7333736785267970048. Current Snapshot: MvccSnapshot[applied={T|T < 7333736785345851392}]
I20260926 22:51:22.365058 27626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:51:22.365136 27626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:51:22.365820 27626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Sending DeleteTablet for 3 replicas of tablet e12abe023e0d42bdb0ce66b4adb50068
I20260926 22:51:22.366103 28019 tablet_service.cc:1514] Processing DeleteTablet for tablet e12abe023e0d42bdb0ce66b4adb50068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:22 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:41746
I20260926 22:51:22.366139 27751 tablet_service.cc:1514] Processing DeleteTablet for tablet e12abe023e0d42bdb0ce66b4adb50068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:22 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:49234
I20260926 22:51:22.366322 28122 tablet_replica.cc:330] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: stopping tablet replica
I20260926 22:51:22.366324 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 27688
I20260926 22:51:22.366334 27885 tablet_service.cc:1514] Processing DeleteTablet for tablet e12abe023e0d42bdb0ce66b4adb50068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:22 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46182
I20260926 22:51:22.366418 28122 raft_consensus.cc:2229] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:51:22.366509 28122 raft_consensus.cc:2258] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:22.366914 28124 tablet_replica.cc:330] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: stopping tablet replica
I20260926 22:51:22.367031 28124 raft_consensus.cc:2229] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:51:22.367040 28122 ts_tablet_manager.cc:1905] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:22.367167 28124 raft_consensus.cc:2258] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:22.367578 28124 ts_tablet_manager.cc:1905] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:22.368629 28122 ts_tablet_manager.cc:1918] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:51:22.368703 28122 log.cc:1196] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/e12abe023e0d42bdb0ce66b4adb50068
I20260926 22:51:22.368947 28122 ts_tablet_manager.cc:1939] T e12abe023e0d42bdb0ce66b4adb50068 P 149fd66f51e743bc802e329050394a89: Deleting consensus metadata
I20260926 22:51:22.369027 28124 ts_tablet_manager.cc:1918] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:51:22.369091 28124 log.cc:1196] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/e12abe023e0d42bdb0ce66b4adb50068
I20260926 22:51:22.369330 28124 ts_tablet_manager.cc:1939] T e12abe023e0d42bdb0ce66b4adb50068 P c26b5143e126414391cc065ab2288b2e: Deleting consensus metadata
I20260926 22:51:22.369309 27613 catalog_manager.cc:4841] TS 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859): tablet e12abe023e0d42bdb0ce66b4adb50068 (table test-table [id=4044005d78f346f1b57679f66fffe110]) successfully deleted
I20260926 22:51:22.369699 27613 catalog_manager.cc:4841] TS c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319): tablet e12abe023e0d42bdb0ce66b4adb50068 (table test-table [id=4044005d78f346f1b57679f66fffe110]) successfully deleted
W20260926 22:51:22.371765 27611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:42817 (error 108)
I20260926 22:51:22.372004 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 27822
I20260926 22:51:22.372160 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.372048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42817 (local address 127.0.0.1:49248)
0926 22:51:22.372089 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.372122 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 22:51:22.372972 27611 catalog_manager.cc:4584] TS 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1:42817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e12abe023e0d42bdb0ce66b4adb50068: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42817: connect: Connection refused (error 111)
I20260926 22:51:22.376479 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 27956
I20260926 22:51:22.380649 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 27594
I20260926 22:51:22.384832 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39089
--webserver_interface=127.13.148.62
--webserver_port=35931
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:22.461793 28126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:22.461979 28126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:22.461998 28126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:22.463131 28126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:22.463179 28126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:22.463192 28126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:22.463204 28126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:22.463227 28126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:22.464655 28126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=35931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:22.464871 28126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:22.465073 28126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:22.472836 28132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.472819 28135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:22.472847 28126 server_base.cc:1034] running on GCE node
W20260926 22:51:22.472824 28133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:22.473237 28126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:22.473519 28126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:22.474651 28126 hybrid_clock.cc:648] HybridClock initialized: now 1790463082474630 us; error 32 us; skew 500 ppm
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:22.476889 28126 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:51:22.477300 28126 webserver.cc:460] Webserver started at http://127.13.148.62:35931/ using document root <none> and password file <none>
I20260926 22:51:22.477478 28126 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:22.477525 28126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:22.478785 28126 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:22.479319 28142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.479457 28126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:51:22.479533 28126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "ca576ae926344e978d3cb1f2bec9f6a6"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "50c8fd20756112ed9e8e0ed44f2bb71f"
server_key_iv: "6f17e269e03ebb94e918b626460855fa"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.479904 28126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:22.496630 28126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:22.498313 28126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:22.498435 28126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:22.502096 28126 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39089
I20260926 22:51:22.502146 28194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39089 every 8 connection(s)
I20260926 22:51:22.502475 28126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:51:22.503944 28195 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:51:22.504534 28195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Bootstrap starting.
I20260926 22:51:22.508883 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 28126
I20260926 22:51:22.509114 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:42817
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:39089
--webserver_port=42071
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:22.512301 28195 log.cc:824] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:22.514072 28195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:51:22.514317 28195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Bootstrap complete.
I20260926 22:51:22.516377 28195 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:22.516594 28195 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca576ae926344e978d3cb1f2bec9f6a6, State: Initialized, Role: FOLLOWER
I20260926 22:51:22.516747 28195 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:22.516821 28195 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:51:22.516872 28195 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:51:22.516919 28195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:51:22.517901 28195 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:22.517995 28195 leader_election.cc:302] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca576ae926344e978d3cb1f2bec9f6a6; no voters:
I20260926 22:51:22.518138 28195 leader_election.cc:288] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 22:51:22.518186 28199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:51:22.518293 28199 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 2 LEADER]: Becoming Leader. State: Replica: ca576ae926344e978d3cb1f2bec9f6a6, State: Running, Role: LEADER
I20260926 22:51:22.518393 28195 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:22.518380 28199 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:22.518620 28199 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca576ae926344e978d3cb1f2bec9f6a6. Latest consensus state: current_term: 2 leader_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } } }
I20260926 22:51:22.518693 28199 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:22.518800 28200 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } } }
I20260926 22:51:22.518869 28203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:51:22.518880 28200 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:22.519636 28203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:51:22.519753 28203 catalog_manager.cc:1259] Loaded cluster ID: cb81e8c1abe24f1fba0a155be30314c9
I20260926 22:51:22.519966 28203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:51:22.520190 28203 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 22:51:22.522913 28203 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 22:51:22.524580 28203 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:51:22.524737 28203 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Loaded TSK: 0
I20260926 22:51:22.524916 28203 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:51:22.600059 28197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:22.600258 28197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:22.600289 28197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:22.601441 28197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:22.601480 28197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:22.601513 28197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:51:22.603039 28197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:42817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=42071
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:22.603255 28197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:22.603448 28197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:22.608945 28225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.608951 28223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.609012 28222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:22.609458 28197 server_base.cc:1034] running on GCE node
I20260926 22:51:22.609634 28197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:22.609869 28197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:22.611040 28197 hybrid_clock.cc:648] HybridClock initialized: now 1790463082611020 us; error 36 us; skew 500 ppm
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:22.613349 28197 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:51:22.613793 28197 webserver.cc:460] Webserver started at http://127.13.148.1:42071/ using document root <none> and password file <none>
I20260926 22:51:22.613998 28197 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:22.614053 28197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:22.615285 28197 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:51:22.615895 28232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.616118 28197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:51:22.616197 28197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "1e802fa25d224c5cb412a67835daacf0"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "a2604e20a0c6cbb07670486c60388bd0"
server_key_iv: "6d606a414ec88645c0ffe2f48cb4e450"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.616530 28197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:22.632469 28197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:22.634555 28197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:22.634687 28197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:22.634891 28197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:22.635288 28239 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:51:22.637109 28197 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:51:22.637154 28197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 22:51:22.637187 28197 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:51:22.637807 28197 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:51:22.637840 28197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260926 22:51:22.637912 28239 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Bootstrap starting.
I20260926 22:51:22.644150 28197 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:42817
I20260926 22:51:22.644335 28346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:42817 every 8 connection(s)
I20260926 22:51:22.644554 28197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:51:22.648088 28239 log.cc:824] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:22.650204 28239 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:51:22.650441 28239 tablet_bootstrap.cc:492] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Bootstrap complete.
I20260926 22:51:22.651175 28239 ts_tablet_manager.cc:1397] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:22.652339 28239 raft_consensus.cc:348] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.652632 28239 raft_consensus.cc:729] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e802fa25d224c5cb412a67835daacf0, State: Initialized, Role: FOLLOWER
I20260926 22:51:22.652802 28239 consensus_queue.cc:260] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } }
I20260926 22:51:22.653080 28239 ts_tablet_manager.cc:1428] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:51:22.653793 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 28197
I20260926 22:51:22.654083 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:46319
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:39089
--webserver_port=40895
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:22.656769 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.645229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:45541 (local address 127.13.148.62:39089)
0926 22:51:22.646522 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.646529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.646564 (+ 35us) server_negotiation.cc:407] Connection header received
0926 22:51:22.646664 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.646669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.646708 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.646819 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:22.647422 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.648332 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.649876 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.650449 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.651342 (+ 893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.653144 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:22.653148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:22.653148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:22.653160 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:22.654926 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.655407 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.655410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.655411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.655523 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.655793 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.655794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.655794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.656072 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:22.656148 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.656478 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.656603 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":84,"threads_started":1}
I20260926 22:51:22.656769 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.645141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:45541)
0926 22:51:22.645838 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.645873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.646283 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.646864 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.646867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.647130 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:22.647306 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.647310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.648446 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.648447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.649774 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.649777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.650571 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.650572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.650721 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.651271 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:22.651280 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:22.653055 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:22.655009 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.655016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.655017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.655305 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.655594 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.655595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.655595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.655734 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.656145 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:22.656152 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:22.656303 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.656410 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.656572 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":84,"threads_started":1}
I20260926 22:51:22.657328 28347 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39089
I20260926 22:51:22.657415 28347 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:22.657573 28347 heartbeater.cc:511] Master 127.13.148.62:39089 requested a full tablet report, sending...
I20260926 22:51:22.658247 28159 ts_manager.cc:194] Registered new tserver with Master: 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1:42817)
I20260926 22:51:22.659243 28159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:45541
I20260926 22:51:22.660094 28347 heartbeater.cc:503] Master 127.13.148.62:39089 was elected leader, sending a full tablet report...
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:51:22.669273 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.658756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42817 (local address 127.0.0.1:49250)
0926 22:51:22.659187 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.659228 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.659297 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.659581 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.659585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.659880 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:22.659929 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.659931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.661093 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.661094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.661642 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.661644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.662271 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.662272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.662326 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.665548 (+ 3222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:22.665552 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:22.666723 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:22.668381 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.668387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.668388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.668562 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.668739 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.668739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.668739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.668783 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.669037 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:22.669041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:22.669078 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.669133 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.669209 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":68,"threads_started":1}
I20260926 22:51:22.669360 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.658956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.13.148.1:42817)
0926 22:51:22.659422 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.659426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.659435 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:51:22.659465 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.659467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.659483 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.659575 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:22.660020 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.661020 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.661747 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.662186 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.662253 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.666797 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:22.666804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:22.666804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:22.666811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:22.668315 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.668627 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.668630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.668630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.668687 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.668828 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.668828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.668829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.668981 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:22.669050 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.669183 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.669269 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":55,"threads_started":1}
I20260926 22:51:22.669821 28281 tablet_service.cc:1514] Processing DeleteTablet for tablet e12abe023e0d42bdb0ce66b4adb50068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:22 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:49250
I20260926 22:51:22.670261 28356 tablet_replica.cc:330] stopping tablet replica
I20260926 22:51:22.670356 28356 raft_consensus.cc:2229] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:51:22.670418 28356 raft_consensus.cc:2258] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:22.670687 28356 ts_tablet_manager.cc:1905] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:22.672571 28356 ts_tablet_manager.cc:1918] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:51:22.672632 28356 log.cc:1196] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/e12abe023e0d42bdb0ce66b4adb50068
I20260926 22:51:22.672822 28356 ts_tablet_manager.cc:1939] T e12abe023e0d42bdb0ce66b4adb50068 P 1e802fa25d224c5cb412a67835daacf0: Deleting consensus metadata
I20260926 22:51:22.673223 28144 catalog_manager.cc:4841] TS 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1:42817): tablet e12abe023e0d42bdb0ce66b4adb50068 (table test-table [id=4044005d78f346f1b57679f66fffe110]) successfully deleted
W20260926 22:51:22.734812 28353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:22.735001 28353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:22.735033 28353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:22.736178 28353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:22.736217 28353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:22.736243 28353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:51:22.737635 28353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:46319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=40895
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:22.737867 28353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:22.738070 28353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:22.742584 28362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.742599 28364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.742617 28361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:22.742784 28353 server_base.cc:1034] running on GCE node
I20260926 22:51:22.742965 28353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:22.743160 28353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:22.744300 28353 hybrid_clock.cc:648] HybridClock initialized: now 1790463082744288 us; error 29 us; skew 500 ppm
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:22.746651 28353 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:51:22.747026 28353 webserver.cc:460] Webserver started at http://127.13.148.2:40895/ using document root <none> and password file <none>
I20260926 22:51:22.747201 28353 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:22.747251 28353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:22.748447 28353 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:22.748943 28371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.749082 28353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.749179 28353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "c26b5143e126414391cc065ab2288b2e"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "7ce0c5aca26bbc68702b952d777fb20a"
server_key_iv: "7ecb5bd2222fbd8cf74d51f9bba8633b"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.749490 28353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:22.786338 28353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:22.788026 28353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:22.788133 28353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:22.788334 28353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:22.788579 28353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:22.788607 28353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.788627 28353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:22.788643 28353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.794219 28353 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:46319
I20260926 22:51:22.794256 28484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:46319 every 8 connection(s)
I20260926 22:51:22.794576 28353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:51:22.799801 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 28353
I20260926 22:51:22.800088 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:34859
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:39089
--webserver_port=33201
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:22.803259 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.795226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:55605 (local address 127.13.148.62:39089)
0926 22:51:22.795292 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.795293 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.795485 (+ 192us) server_negotiation.cc:407] Connection header received
0926 22:51:22.795987 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.795989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.796000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.796055 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:22.796947 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.797433 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.798389 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.798584 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.798625 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.800680 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:22.800682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:22.800682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:22.800690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:22.802002 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.802436 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.802437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.802437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.802479 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.802715 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.802715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.802715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.802805 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:22.802846 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.803126 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.803196 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:51:22.803498 28487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.795206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:55605)
0926 22:51:22.795366 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.795407 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.795879 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.796130 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.796133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.796686 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:22.796877 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.796881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.797493 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.797494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.798324 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.798327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.798635 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.798636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.798730 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.799166 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:22.799174 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:22.800589 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:22.802071 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.802077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.802079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.802370 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.802526 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.802527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.802527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.802661 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.802852 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:22.802857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:22.803009 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.803117 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.803271 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260926 22:51:22.804054 28485 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39089
I20260926 22:51:22.804141 28485 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:22.804314 28485 heartbeater.cc:511] Master 127.13.148.62:39089 requested a full tablet report, sending...
I20260926 22:51:22.804693 28159 ts_manager.cc:194] Registered new tserver with Master: c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319)
I20260926 22:51:22.805181 28159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:55605
W20260926 22:51:22.878680 28488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:22.878860 28488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:22.878881 28488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:22.879935 28488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:22.879964 28488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:22.879988 28488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:51:22.881337 28488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:34859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=33201
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:22.881510 28488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:22.881697 28488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:22.886309 28496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.886317 28494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:22.886343 28493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:22.886562 28488 server_base.cc:1034] running on GCE node
I20260926 22:51:22.886798 28488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:22.887032 28488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:22.888180 28488 hybrid_clock.cc:648] HybridClock initialized: now 1790463082888159 us; error 34 us; skew 500 ppm
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:22.890471 28488 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:51:22.890869 28488 webserver.cc:460] Webserver started at http://127.13.148.3:33201/ using document root <none> and password file <none>
I20260926 22:51:22.891067 28488 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:22.891126 28488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:22.892323 28488 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:51:22.892838 28503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.892997 28488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.893074 28488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "149fd66f51e743bc802e329050394a89"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "60cfe2b2031424ebdd78dc2addd3f0dc"
server_key_iv: "90b9fb5033d8a57e049a820a251334c2"
server_key_version: "encryptionkey@0"
I20260926 22:51:22.893392 28488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:22.908857 28488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:22.910918 28488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:22.911051 28488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:22.911250 28488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:22.911522 28488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:22.911554 28488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.911581 28488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:22.911604 28488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:22.916908 28488 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:34859
I20260926 22:51:22.916950 28616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:34859 every 8 connection(s)
I20260926 22:51:22.917271 28488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:51:22 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463082, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:22.925107 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 28488
I20260926 22:51:22.925531 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.917939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:51745 (local address 127.13.148.62:39089)
0926 22:51:22.917982 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.917983 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.918207 (+ 224us) server_negotiation.cc:407] Connection header received
0926 22:51:22.918721 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.918723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.918756 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.918801 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:22.919749 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.920240 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.921176 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.921380 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.921435 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.923339 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:22.923340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:22.923341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:22.923347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:22.924372 (+ 1025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.924701 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.924702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.924702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.924747 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.924930 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.924931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.924931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.925057 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:22.925122 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.925362 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.925444 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:51:22.925704 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.917931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:51745)
0926 22:51:22.918057 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.918109 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.918611 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.918880 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.918884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.919465 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:22.919654 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.919659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.920318 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.920319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.921093 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.921095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.921448 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.921449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.921559 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.921962 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:22.921970 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:22.923247 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:22.924435 (+ 1188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.924440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.924442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.924647 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.924789 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:22.924789 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:22.924790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:22.924874 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:22.925110 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:22.925113 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:22.925229 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.925328 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.925471 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260926 22:51:22.926115 28617 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39089
I20260926 22:51:22.926200 28617 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:22.926367 28617 heartbeater.cc:511] Master 127.13.148.62:39089 requested a full tablet report, sending...
I20260926 22:51:22.926728 28159 ts_manager.cc:194] Registered new tserver with Master: 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859)
I20260926 22:51:22.927058 28159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:51745
I20260926 22:51:22.930943 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.925450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.13.148.62:39089)
0926 22:51:22.925548 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.925549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.925663 (+ 114us) server_negotiation.cc:407] Connection header received
0926 22:51:22.925780 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.925781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.925793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.925858 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:22.926518 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.927076 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.927831 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.928012 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.928063 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.928975 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:22.928976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:22.928976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:22.928982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:22.930198 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.930447 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.930448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.930448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.930491 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:22.930605 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:22.930605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:22.930606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:22.930686 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:22.930784 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.930826 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.930883 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":56,"threads_started":1}
I20260926 22:51:22.931547 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:51:22.933809 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.931839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.13.148.62:39089)
0926 22:51:22.931870 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.931871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.931963 (+ 92us) server_negotiation.cc:407] Connection header received
0926 22:51:22.932025 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.932026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.932051 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.932098 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:51:22.932163 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.932594 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.933244 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.933417 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.933462 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.933549 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:51:22.933646 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:51:22.933699 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.933711 (+ 12us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.933748 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 22:51:22.934062 28159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:51:22.935572 28159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:51:22.937397 28281 tablet_service.cc:1467] Processing CreateTablet for tablet f3d2a6b74bfe4f789196c5e39f7d29cd (DEFAULT_TABLE table=test-table [id=661da2a2ca4649c6a2fc510a4f9ee700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:22.937587 28281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d2a6b74bfe4f789196c5e39f7d29cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:22.939047 28239 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Bootstrap starting.
I20260926 22:51:22.939358 28239 tablet_bootstrap.cc:654] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:22.939910 28239 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: No bootstrap required, opened a new log
I20260926 22:51:22.939962 28239 ts_tablet_manager.cc:1397] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:22.940291 28239 raft_consensus.cc:348] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:22.940367 28239 raft_consensus.cc:374] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:22.940411 28239 raft_consensus.cc:729] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e802fa25d224c5cb412a67835daacf0, State: Initialized, Role: FOLLOWER
I20260926 22:51:22.940449 28239 consensus_queue.cc:260] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:22.940524 28239 ts_tablet_manager.cc:1428] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:51:22.941188 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.937147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34859 (local address 127.0.0.1:41756)
0926 22:51:22.937181 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.937191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.937225 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.937713 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.937714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.937715 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:22.937780 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.937782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.938950 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.938951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.939936 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.939939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.940977 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.940978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.941048 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.941067 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.941094 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.941135 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 22:51:22.941496 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.937320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.13.148.3:34859)
0926 22:51:22.937508 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.937512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.937523 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:51:22.937556 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.937559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.937594 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.937702 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:22.937846 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.938855 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.940197 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.940886 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.941333 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.941393 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.941413 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.941444 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260926 22:51:22.942052 28551 tablet_service.cc:1467] Processing CreateTablet for tablet f3d2a6b74bfe4f789196c5e39f7d29cd (DEFAULT_TABLE table=test-table [id=661da2a2ca4649c6a2fc510a4f9ee700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:22.942358 28551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d2a6b74bfe4f789196c5e39f7d29cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:22.942516 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.937235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46319 (local address 127.0.0.1:46198)
0926 22:51:22.937362 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:22.937372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:22.937415 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:22.939543 (+ 2128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:22.939544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:22.939546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:22.939587 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.939588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.940745 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.940746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.941593 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:22.941596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.942284 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.942285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:22.942348 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.942352 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:51:22.942406 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 22:51:22.942445 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260926 22:51:22.942667 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:22.939118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.13.148.2:46319)
0926 22:51:22.939302 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:22.939307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:22.939317 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:51:22.939353 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:22.939355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:22.939370 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:22.939452 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:22.939645 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.940638 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.941664 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:22.942177 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:22.942242 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:22.942314 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:22.942469 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 22:51:22.942520 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260926 22:51:22.943186 28419 tablet_service.cc:1467] Processing CreateTablet for tablet f3d2a6b74bfe4f789196c5e39f7d29cd (DEFAULT_TABLE table=test-table [id=661da2a2ca4649c6a2fc510a4f9ee700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:22.943449 28419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d2a6b74bfe4f789196c5e39f7d29cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:22.945089 28627 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Bootstrap starting.
I20260926 22:51:22.945761 28627 tablet_bootstrap.cc:654] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:22.946228 28628 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Bootstrap starting.
I20260926 22:51:22.946316 28627 log.cc:824] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:22.946822 28628 tablet_bootstrap.cc:654] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:22.947014 28627 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: No bootstrap required, opened a new log
I20260926 22:51:22.947067 28627 ts_tablet_manager.cc:1397] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:22.947258 28628 log.cc:824] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:22.947885 28628 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: No bootstrap required, opened a new log
I20260926 22:51:22.947939 28628 ts_tablet_manager.cc:1397] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:22.948565 28627 raft_consensus.cc:348] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:22.948721 28627 raft_consensus.cc:374] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:22.948755 28627 raft_consensus.cc:729] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149fd66f51e743bc802e329050394a89, State: Initialized, Role: FOLLOWER
I20260926 22:51:22.948856 28627 consensus_queue.cc:260] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:22.949105 28627 ts_tablet_manager.cc:1428] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:51:22.949115 28617 heartbeater.cc:503] Master 127.13.148.62:39089 was elected leader, sending a full tablet report...
I20260926 22:51:22.949599 28628 raft_consensus.cc:348] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:22.949772 28628 raft_consensus.cc:374] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:22.949819 28628 raft_consensus.cc:729] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26b5143e126414391cc065ab2288b2e, State: Initialized, Role: FOLLOWER
I20260926 22:51:22.949929 28628 consensus_queue.cc:260] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:22.950182 28628 ts_tablet_manager.cc:1428] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:51:22.950207 28485 heartbeater.cc:503] Master 127.13.148.62:39089 was elected leader, sending a full tablet report...
W20260926 22:51:23.045094 28486 tablet.cc:2369] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:51:23.089538 28352 raft_consensus.cc:482] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:51:23.089668 28352 raft_consensus.cc:504] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:23.089990 28352 leader_election.cc:288] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859), c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319)
I20260926 22:51:23.092659 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.090173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34859 (local address 127.13.148.1:42121)
0926 22:51:23.090224 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.090237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:23.090290 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:23.090452 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:23.090453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:23.090455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:23.090522 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:23.090524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.091132 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.091133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:23.091894 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:23.091896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.092419 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.092419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:23.092485 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:23.092502 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 22:51:23.092548 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:51:23.092587 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 22:51:23.092679 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.090237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42121 (local address 127.13.148.3:34859)
0926 22:51:23.090287 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:23.090288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:23.090299 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:51:23.090389 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:23.090391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:23.090400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:23.090447 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:23.090583 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.091054 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.091981 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.092342 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.092402 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:23.092411 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:23.092563 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 22:51:23.092616 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:51:23.092885 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.090363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:41139 (local address 127.13.148.2:46319)
0926 22:51:23.090410 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:23.090411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:23.090508 (+ 97us) server_negotiation.cc:407] Connection header received
0926 22:51:23.090566 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:23.090567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:23.090576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:23.090625 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:23.090790 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.091284 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.092156 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.092531 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.092588 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:23.092595 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:23.092774 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 22:51:23.092817 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 22:51:23.093006 28571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d2a6b74bfe4f789196c5e39f7d29cd" candidate_uuid: "1e802fa25d224c5cb412a67835daacf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149fd66f51e743bc802e329050394a89" is_pre_election: true
I20260926 22:51:23.093124 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.090292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46319 (local address 127.13.148.1:41139)
0926 22:51:23.090444 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.090454 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:23.090515 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:23.090632 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:23.090633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:23.090634 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:23.090727 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:23.090728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.091350 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.091351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:23.092075 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:23.092077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.092619 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.092620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:23.092689 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:23.092692 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:51:23.093001 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 22:51:23.093044 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260926 22:51:23.093169 28571 raft_consensus.cc:2454] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e802fa25d224c5cb412a67835daacf0 in term 0.
I20260926 22:51:23.093415 28439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d2a6b74bfe4f789196c5e39f7d29cd" candidate_uuid: "1e802fa25d224c5cb412a67835daacf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c26b5143e126414391cc065ab2288b2e" is_pre_election: true
I20260926 22:51:23.093552 28439 raft_consensus.cc:2454] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e802fa25d224c5cb412a67835daacf0 in term 0.
I20260926 22:51:23.093747 28236 leader_election.cc:302] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149fd66f51e743bc802e329050394a89, 1e802fa25d224c5cb412a67835daacf0; no voters:
I20260926 22:51:23.093873 28352 raft_consensus.cc:2781] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:51:23.093940 28352 raft_consensus.cc:482] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:51:23.093955 28352 raft_consensus.cc:3037] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:23.094630 28352 raft_consensus.cc:504] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:23.094734 28352 leader_election.cc:288] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 1 election: Requested vote from peers 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859), c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319)
I20260926 22:51:23.094892 28571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d2a6b74bfe4f789196c5e39f7d29cd" candidate_uuid: "1e802fa25d224c5cb412a67835daacf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149fd66f51e743bc802e329050394a89"
I20260926 22:51:23.094904 28439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d2a6b74bfe4f789196c5e39f7d29cd" candidate_uuid: "1e802fa25d224c5cb412a67835daacf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c26b5143e126414391cc065ab2288b2e"
I20260926 22:51:23.094957 28571 raft_consensus.cc:3037] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:23.094976 28439 raft_consensus.cc:3037] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:23.095651 28571 raft_consensus.cc:2454] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e802fa25d224c5cb412a67835daacf0 in term 1.
I20260926 22:51:23.095674 28439 raft_consensus.cc:2454] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e802fa25d224c5cb412a67835daacf0 in term 1.
I20260926 22:51:23.095826 28236 leader_election.cc:302] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e802fa25d224c5cb412a67835daacf0, c26b5143e126414391cc065ab2288b2e; no voters:
I20260926 22:51:23.095914 28352 raft_consensus.cc:2781] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:23.096077 28352 raft_consensus.cc:686] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 LEADER]: Becoming Leader. State: Replica: 1e802fa25d224c5cb412a67835daacf0, State: Running, Role: LEADER
I20260926 22:51:23.096136 28352 consensus_queue.cc:237] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:23.096860 28159 catalog_manager.cc:5495] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "1e802fa25d224c5cb412a67835daacf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } health_report { overall_health: HEALTHY } } }
I20260926 22:51:23.120224 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.117853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.13.148.1:42817)
0926 22:51:23.117904 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:23.117906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:23.117914 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:51:23.117985 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:23.117986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:23.118004 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:23.118061 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:51:23.118146 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.118742 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.119515 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:23.119760 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:23.119806 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:23.119951 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:51:23.120051 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:51:23.120100 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:23.120131 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 22:51:23.120148 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:51:23.121614 28439 raft_consensus.cc:1261] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 1 FOLLOWER]: Refusing update from remote peer 1e802fa25d224c5cb412a67835daacf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:23.121628 28571 raft_consensus.cc:1261] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Refusing update from remote peer 1e802fa25d224c5cb412a67835daacf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:23.121956 28634 consensus_queue.cc:1035] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:23.122071 28352 consensus_queue.cc:1035] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:23.124548 28639 mvcc.cc:205] Tried to move back new op lower bound from 7333736788463796224 to 7333736788362211328. Current Snapshot: MvccSnapshot[applied={T|T < 7333736788463796224}]
I20260926 22:51:23.124678 28637 mvcc.cc:205] Tried to move back new op lower bound from 7333736788463796224 to 7333736788362211328. Current Snapshot: MvccSnapshot[applied={T|T < 7333736788463796224 or (T in {7333736788463796224})}]
I20260926 22:51:23.125401 28638 mvcc.cc:205] Tried to move back new op lower bound from 7333736788463796224 to 7333736788362211328. Current Snapshot: MvccSnapshot[applied={T|T < 7333736788463796224}]
I20260926 22:51:23.126595 28158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:51:23.126682 28158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:51:23.127313 28158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Sending DeleteTablet for 3 replicas of tablet f3d2a6b74bfe4f789196c5e39f7d29cd
I20260926 22:51:23.127575 28281 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d2a6b74bfe4f789196c5e39f7d29cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:23 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:49250
I20260926 22:51:23.127677 28356 tablet_replica.cc:330] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: stopping tablet replica
I20260926 22:51:23.127681 28551 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d2a6b74bfe4f789196c5e39f7d29cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:23 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:41756
I20260926 22:51:23.127727 28356 raft_consensus.cc:2229] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:51:23.127802 28356 raft_consensus.cc:2258] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:23.127920 28647 tablet_replica.cc:330] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: stopping tablet replica
I20260926 22:51:23.128003 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 28197
I20260926 22:51:23.128021 28419 tablet_service.cc:1514] Processing DeleteTablet for tablet f3d2a6b74bfe4f789196c5e39f7d29cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:23 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:46198
I20260926 22:51:23.128851 28648 tablet_replica.cc:330] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: stopping tablet replica
I20260926 22:51:23.128952 28648 raft_consensus.cc:2229] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:51:23.129056 28648 raft_consensus.cc:2258] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:23.129408 28648 ts_tablet_manager.cc:1905] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:23.130834 28648 ts_tablet_manager.cc:1918] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:51:23.130898 28648 log.cc:1196] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/f3d2a6b74bfe4f789196c5e39f7d29cd
I20260926 22:51:23.131099 28648 ts_tablet_manager.cc:1939] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Deleting consensus metadata
I20260926 22:51:23.131438 28146 catalog_manager.cc:4841] TS c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319): tablet f3d2a6b74bfe4f789196c5e39f7d29cd (table test-table [id=661da2a2ca4649c6a2fc510a4f9ee700]) successfully deleted
W20260926 22:51:23.133028 28144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:42817 (error 108)
I20260926 22:51:23.133458 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.133353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42817 (local address 127.0.0.1:49260)
0926 22:51:23.133400 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.133433 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:51:23.133591 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 28353
W20260926 22:51:23.133572 28144 catalog_manager.cc:4584] TS 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1:42817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3d2a6b74bfe4f789196c5e39f7d29cd: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42817: connect: Connection refused (error 111)
I20260926 22:51:23.137801 28647 raft_consensus.cc:2229] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:51:23.137945 28647 raft_consensus.cc:2258] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:23.138375 28647 ts_tablet_manager.cc:1905] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:23.138443 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 28488
I20260926 22:51:23.142014 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.141914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34859 (local address 127.0.0.1:41772)
0926 22:51:23.141957 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.141980 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.3:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 22:51:23.142194 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 28126
I20260926 22:51:23.146489 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39089
--webserver_interface=127.13.148.62
--webserver_port=35931
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:23.232595 28650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:23.232862 28650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:23.232895 28650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:23.234558 28650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:23.234602 28650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:23.234615 28650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:23.234627 28650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:23.234643 28650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:23.236553 28650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=35931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:23.236800 28650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:23.237077 28650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:23.242365 28656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:23.242378 28658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:23.242386 28655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:23.243023 28650 server_base.cc:1034] running on GCE node
I20260926 22:51:23.243291 28650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:23.243618 28650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:23.244781 28650 hybrid_clock.cc:648] HybridClock initialized: now 1790463083244763 us; error 33 us; skew 500 ppm
Sep 26 22:51:23 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463083, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:23.247258 28650 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:51:23.247710 28650 webserver.cc:460] Webserver started at http://127.13.148.62:35931/ using document root <none> and password file <none>
I20260926 22:51:23.247936 28650 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:23.247978 28650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:23.249567 28650 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:51:23.250325 28665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:23.250459 28650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:23.250542 28650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "ca576ae926344e978d3cb1f2bec9f6a6"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "50c8fd20756112ed9e8e0ed44f2bb71f"
server_key_iv: "6f17e269e03ebb94e918b626460855fa"
server_key_version: "encryptionkey@0"
I20260926 22:51:23.250950 28650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:23.267782 28650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:23.269862 28650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:23.269973 28650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:23.273630 28650 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39089
I20260926 22:51:23.273679 28717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39089 every 8 connection(s)
I20260926 22:51:23.274041 28650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:51:23.275516 28718 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:51:23.276094 28718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Bootstrap starting.
I20260926 22:51:23.280689 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 28650
I20260926 22:51:23.280993 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:42817
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:39089
--webserver_port=42071
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:23.284399 28718 log.cc:824] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:23.286636 28718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:51:23.286885 28718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6: Bootstrap complete.
I20260926 22:51:23.288909 28718 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:23.289122 28718 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca576ae926344e978d3cb1f2bec9f6a6, State: Initialized, Role: FOLLOWER
I20260926 22:51:23.289285 28718 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:23.289361 28718 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:51:23.289407 28718 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:51:23.289453 28718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 2 FOLLOWER]: Advancing to term 3
I20260926 22:51:23.290432 28718 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:23.290524 28718 leader_election.cc:302] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca576ae926344e978d3cb1f2bec9f6a6; no voters:
I20260926 22:51:23.290668 28718 leader_election.cc:288] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 22:51:23.290711 28722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 22:51:23.290822 28722 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [term 3 LEADER]: Becoming Leader. State: Replica: ca576ae926344e978d3cb1f2bec9f6a6, State: Running, Role: LEADER
I20260926 22:51:23.290923 28718 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:23.290910 28722 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } }
I20260926 22:51:23.291196 28723 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } } }
I20260926 22:51:23.291203 28724 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca576ae926344e978d3cb1f2bec9f6a6. Latest consensus state: current_term: 3 leader_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca576ae926344e978d3cb1f2bec9f6a6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39089 } } }
I20260926 22:51:23.291286 28723 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:23.291301 28724 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca576ae926344e978d3cb1f2bec9f6a6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:23.291460 28727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:51:23.292246 28727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:51:23.292352 28727 catalog_manager.cc:1259] Loaded cluster ID: cb81e8c1abe24f1fba0a155be30314c9
I20260926 22:51:23.292387 28727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 22:51:23.292876 28727 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f66758a59ad google::LogMessage::Fail() at ??:0
@ 0x7f66758a888a google::LogMessage::SendToLog() at ??:0
@ 0x7f66758a54de google::LogMessage::Flush() at ??:0
@ 0x7f66758a6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f667d7c60f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f667d7cf8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f66777730d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6677768954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f66785cc6db start_thread at ??:0
@ 0x7f66743a361f clone at ??:0
W20260926 22:51:23.373433 28720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:23.373616 28720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:23.373638 28720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:23.374766 28720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:23.374807 28720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:23.374832 28720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:51:23.376214 28720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:42817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=42071
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:23.376403 28720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:23.376601 28720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:23.381170 28754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:23.381207 28755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:23.381160 28757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:23.381522 28720 server_base.cc:1034] running on GCE node
I20260926 22:51:23.381721 28720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:23.382023 28720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:23.383203 28720 hybrid_clock.cc:648] HybridClock initialized: now 1790463083383183 us; error 30 us; skew 500 ppm
Sep 26 22:51:23 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463083, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:23.386075 28720 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:51:23.386616 28720 webserver.cc:460] Webserver started at http://127.13.148.1:42071/ using document root <none> and password file <none>
I20260926 22:51:23.386866 28720 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:23.386935 28720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:23.388641 28720 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:51:23.389380 28764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:23.389549 28720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:51:23.389645 28720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "1e802fa25d224c5cb412a67835daacf0"
format_stamp: "Formatted at 2026-09-26 22:51:21 on dist-test-slave-98hb"
server_key: "a2604e20a0c6cbb07670486c60388bd0"
server_key_iv: "6d606a414ec88645c0ffe2f48cb4e450"
server_key_version: "encryptionkey@0"
I20260926 22:51:23.390103 28720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:23.406971 28720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:23.409255 28720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:23.409418 28720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:23.409713 28720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:23.410243 28771 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:51:23.411588 28720 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:51:23.411648 28720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 22:51:23.411680 28720 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:51:23.412549 28720 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:51:23.412593 28720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 22:51:23.412649 28771 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Bootstrap starting.
I20260926 22:51:23.419608 28720 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:42817
I20260926 22:51:23.419770 28878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:42817 every 8 connection(s)
I20260926 22:51:23.420048 28720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:51:23.422412 28771 log.cc:824] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:23.424311 28771 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:51:23.424544 28771 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Bootstrap complete.
I20260926 22:51:23.425271 28771 ts_tablet_manager.cc:1397] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260926 22:51:23.426281 28771 raft_consensus.cc:348] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:23.426471 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 28720
I20260926 22:51:23.426501 28771 raft_consensus.cc:729] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e802fa25d224c5cb412a67835daacf0, State: Initialized, Role: FOLLOWER
I20260926 22:51:23.426652 28771 consensus_queue.cc:260] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:23.426748 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:46319
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:39089
--webserver_port=40895
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:51:23.426901 28771 ts_tablet_manager.cc:1428] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
W20260926 22:51:23.495283 28881 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:51:23.420736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:38695)
0926 22:51:23.420944 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.421008 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:23.421634 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:23.495071 (+ 73437us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
W20260926 22:51:23.495512 28879 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:39089 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:39089: Client connection negotiation failed: client connection to 127.13.148.62:39089: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 22:51:23.495851 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.495742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:56353)
0926 22:51:23.495785 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.495822 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 22:51:23.496193 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.496114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:37423)
0926 22:51:23.496153 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.496177 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 22:51:23.496297 28879 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:51:23.506883 28884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:23.507059 28884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:23.507081 28884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:23.508136 28884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:23.508174 28884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:23.508198 28884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:51:23.509528 28884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:46319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=40895
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:23.509698 28884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:23.509918 28884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:23.514398 28890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:23.514389 28892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:23.514398 28889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:23.514596 28884 server_base.cc:1034] running on GCE node
I20260926 22:51:23.514793 28884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:23.515014 28884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:23.516139 28884 hybrid_clock.cc:648] HybridClock initialized: now 1790463083516126 us; error 30 us; skew 500 ppm
Sep 26 22:51:23 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463083, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:23.518389 28884 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:51:23.518770 28884 webserver.cc:460] Webserver started at http://127.13.148.2:40895/ using document root <none> and password file <none>
I20260926 22:51:23.518950 28884 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:23.518987 28884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:23.520155 28884 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:51:23.520649 28899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:23.520809 28884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:23.520877 28884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "c26b5143e126414391cc065ab2288b2e"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "7ce0c5aca26bbc68702b952d777fb20a"
server_key_iv: "7ecb5bd2222fbd8cf74d51f9bba8633b"
server_key_version: "encryptionkey@0"
I20260926 22:51:23.521183 28884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:23.535112 28884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:23.536722 28884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:23.536840 28884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:23.537039 28884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:23.537303 28884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:23.537336 28884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:23.537362 28884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:23.537388 28884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:23.542651 28884 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:46319
I20260926 22:51:23.542711 29012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:46319 every 8 connection(s)
I20260926 22:51:23.542996 28884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:51:23.543905 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.543551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:51495)
0926 22:51:23.543701 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.543771 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
W20260926 22:51:23.544271 29013 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:39089 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:39089: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
I20260926 22:51:23.544490 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.544412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:35269)
0926 22:51:23.544448 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.544472 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 22:51:23.544691 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.544636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:44469)
0926 22:51:23.544663 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.544680 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 22:51:23.544766 29013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 22:51:23.552030 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 28884
I20260926 22:51:23.552264 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:34859
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:39089
--webserver_port=33201
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:23.631430 29016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:23.631621 29016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:23.631642 29016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:23.632741 29016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:23.632779 29016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:23.632807 29016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:51:23.634223 29016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:34859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=33201
--tserver_master_addrs=127.13.148.62:39089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:23.634426 29016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:23.634634 29016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:23.639278 29021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:23.639284 29022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:23.639380 29024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:23.639436 29016 server_base.cc:1034] running on GCE node
I20260926 22:51:23.639618 29016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:23.639801 29016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:23.640926 29016 hybrid_clock.cc:648] HybridClock initialized: now 1790463083640919 us; error 25 us; skew 500 ppm
Sep 26 22:51:23 dist-test-slave-98hb krb5kdc[27581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463083, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:23.643203 29016 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:51:23.643573 29016 webserver.cc:460] Webserver started at http://127.13.148.3:33201/ using document root <none> and password file <none>
I20260926 22:51:23.643752 29016 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:23.643793 29016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:23.644961 29016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:51:23.645560 29031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:23.645702 29016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:51:23.645803 29016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "149fd66f51e743bc802e329050394a89"
format_stamp: "Formatted at 2026-09-26 22:51:22 on dist-test-slave-98hb"
server_key: "60cfe2b2031424ebdd78dc2addd3f0dc"
server_key_iv: "90b9fb5033d8a57e049a820a251334c2"
server_key_version: "encryptionkey@0"
I20260926 22:51:23.646085 29016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:23.653964 29016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:23.655459 29016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:23.655555 29016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:23.655707 29016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:23.656060 29038 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:51:23.657291 29016 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:51:23.657331 29016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260926 22:51:23.657352 29016 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:51:23.657963 29016 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:51:23.657994 29016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 22:51:23.658047 29038 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Bootstrap starting.
I20260926 22:51:23.659510 29038 log.cc:824] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:23.662220 29038 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:51:23.662492 29038 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Bootstrap complete.
I20260926 22:51:23.662631 29038 ts_tablet_manager.cc:1397] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260926 22:51:23.663458 29016 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:34859
I20260926 22:51:23.663508 29146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:34859 every 8 connection(s)
I20260926 22:51:23.663771 29016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:51:23.664099 29038 raft_consensus.cc:348] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:23.664372 29038 raft_consensus.cc:729] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149fd66f51e743bc802e329050394a89, State: Initialized, Role: FOLLOWER
I20260926 22:51:23.664508 29038 consensus_queue.cc:260] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:23.664714 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.664361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:37949)
0926 22:51:23.664492 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.664565 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260926 22:51:23.664746 29038 ts_tablet_manager.cc:1428] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260926 22:51:23.664884 29147 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:39089 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:39089: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
I20260926 22:51:23.665120 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.665040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:46289)
0926 22:51:23.665076 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.665099 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 22:51:23.665349 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:23.665278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:58541)
0926 22:51:23.665311 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:23.665330 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 22:51:23.665431 29147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 22:51:23.666628 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 29016
I20260926 22:51:24.496773 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:24.496576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:46777)
0926 22:51:24.496679 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:24.496713 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260926 22:51:24.545233 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:24.545037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:57559)
0926 22:51:24.545149 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:24.545184 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260926 22:51:24.665877 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:24.665666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:54499)
0926 22:51:24.665802 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:24.665834 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260926 22:51:24.854796 29155 raft_consensus.cc:482] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:51:24.854910 29155 raft_consensus.cc:504] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:24.855248 29155 leader_election.cc:288] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859), c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319)
Sep 26 22:51:24 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463083, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.2@KRBTEST.COM
Sep 26 22:51:24 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463083, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.3@KRBTEST.COM
I20260926 22:51:24.865681 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:24.855573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:50475 (local address 127.13.148.2:46319)
0926 22:51:24.855744 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:24.855755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:24.855781 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:51:24.856218 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:24.856220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:24.856257 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:24.856378 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:24.856762 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:24.858171 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:24.859587 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:24.860205 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:24.860294 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:24.862635 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:24.862639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:24.862639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:24.862649 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:24.864143 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:24.864502 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:24.864504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:24.864505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:24.864593 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:24.864803 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:24.864804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:24.864805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:24.865091 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:24.865177 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:24.865494 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 22:51:24.865607 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260926 22:51:24.865679 29157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:24.855492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:46319 (local address 127.13.148.1:50475)
0926 22:51:24.855617 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:24.855624 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:24.855682 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:24.856385 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:24.856386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:24.856621 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:24.856666 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:24.856668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:24.858268 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:24.858269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:24.859510 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:24.859513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:24.860282 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:24.860283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:24.860417 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:24.860956 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:24.860966 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:24.862537 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:24.864215 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:24.864221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:24.864223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:24.864433 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:24.864649 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:24.864650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:24.864650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:24.864751 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:24.865171 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:24.865178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:24.865330 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 22:51:24.865422 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 22:51:24.865570 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":59,"thread_start_us":65,"threads_started":1}
I20260926 22:51:24.866178 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:24.855381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:34859 (local address 127.13.148.1:38961)
0926 22:51:24.855423 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:24.855434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:24.855488 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:24.855957 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:24.855960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:24.856332 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:24.856625 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:24.856633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:24.858260 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:24.858260 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:24.859510 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:24.859513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:24.860281 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:24.860282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:24.860417 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:24.860976 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:24.860980 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:24.862875 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:24.864906 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:24.864907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:24.864908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:24.865109 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:24.865357 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:24.865357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:24.865358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:24.865408 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:24.865886 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:24.865887 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:24.865928 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 22:51:24.865989 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 22:51:24.866080 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":54}
I20260926 22:51:24.866304 29156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:24.855469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38961 (local address 127.13.148.3:34859)
0926 22:51:24.855616 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:24.855629 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:24.855649 (+ 20us) server_negotiation.cc:407] Connection header received
0926 22:51:24.855749 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:24.855750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:24.855790 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:24.855924 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:24.856758 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:24.858165 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:24.859588 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:24.860187 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:24.860293 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:24.862979 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:24.862983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:24.862983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:24.862994 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:24.864846 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:24.865184 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:24.865187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:24.865188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:24.865295 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:24.865467 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:24.865468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:24.865468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:24.865780 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:24.865858 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:24.866065 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 22:51:24.866200 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
W20260926 22:51:24.866444 28768 leader_election.cc:341] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319): Not found: Tablet not found: f3d2a6b74bfe4f789196c5e39f7d29cd
I20260926 22:51:24.866832 29101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d2a6b74bfe4f789196c5e39f7d29cd" candidate_uuid: "1e802fa25d224c5cb412a67835daacf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "149fd66f51e743bc802e329050394a89" is_pre_election: true
I20260926 22:51:24.866958 29101 raft_consensus.cc:2454] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e802fa25d224c5cb412a67835daacf0 in term 1.
I20260926 22:51:24.867184 28768 leader_election.cc:302] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149fd66f51e743bc802e329050394a89, 1e802fa25d224c5cb412a67835daacf0; no voters: c26b5143e126414391cc065ab2288b2e
I20260926 22:51:24.867300 29155 raft_consensus.cc:2781] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 22:51:24.867376 29155 raft_consensus.cc:482] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:51:24.867393 29155 raft_consensus.cc:3037] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:51:24.868439 29155 raft_consensus.cc:504] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:24.868551 29155 leader_election.cc:288] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 2 election: Requested vote from peers 149fd66f51e743bc802e329050394a89 (127.13.148.3:34859), c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319)
I20260926 22:51:24.868718 29101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d2a6b74bfe4f789196c5e39f7d29cd" candidate_uuid: "1e802fa25d224c5cb412a67835daacf0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "149fd66f51e743bc802e329050394a89"
W20260926 22:51:24.868813 28768 leader_election.cc:341] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319): Not found: Tablet not found: f3d2a6b74bfe4f789196c5e39f7d29cd
I20260926 22:51:24.868809 29101 raft_consensus.cc:3037] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:51:24.869935 29101 raft_consensus.cc:2454] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e802fa25d224c5cb412a67835daacf0 in term 2.
I20260926 22:51:24.870082 28768 leader_election.cc:302] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 149fd66f51e743bc802e329050394a89, 1e802fa25d224c5cb412a67835daacf0; no voters: c26b5143e126414391cc065ab2288b2e
I20260926 22:51:24.870180 29155 raft_consensus.cc:2781] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:51:24.870306 29155 raft_consensus.cc:686] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [term 2 LEADER]: Becoming Leader. State: Replica: 1e802fa25d224c5cb412a67835daacf0, State: Running, Role: LEADER
I20260926 22:51:24.870390 29155 consensus_queue.cc:237] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:24.870559 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:24.870460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:49343)
0926 22:51:24.870501 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:24.870526 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 22:51:25.310345 29101 raft_consensus.cc:1261] T f3d2a6b74bfe4f789196c5e39f7d29cd P 149fd66f51e743bc802e329050394a89 [term 2 FOLLOWER]: Refusing update from remote peer 1e802fa25d224c5cb412a67835daacf0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 22:51:25.310717 29155 consensus_queue.cc:1035] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 22:51:25.311739 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:25.311482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:58369)
0926 22:51:25.311595 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:25.311689 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":45,"threads_started":1}
W20260926 22:51:25.312462 28768 consensus_peers.cc:487] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0 -> Peer c26b5143e126414391cc065ab2288b2e (127.13.148.2:46319): Couldn't send request to peer c26b5143e126414391cc065ab2288b2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3d2a6b74bfe4f789196c5e39f7d29cd. This is attempt 1: this message will repeat every 5th retry.
I20260926 22:51:25.312552 29163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:25.312350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:51363)
0926 22:51:25.312501 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:25.312528 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 22:51:25.312557 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:25.312460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:56309)
0926 22:51:25.312510 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:25.312531 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:51:25.545848 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:25.545631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:56083)
0926 22:51:25.545773 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:25.545809 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260926 22:51:25.868793 29169 ts_tablet_manager.cc:927] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Initiating tablet copy from peer 1e802fa25d224c5cb412a67835daacf0 (127.13.148.1:42817)
I20260926 22:51:25.868999 29169 tablet_copy_client.cc:316] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: tablet copy: Beginning tablet copy session from remote peer at address 127.13.148.1:42817
Sep 26 22:51:25 dist-test-slave-98hb krb5kdc[27581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463083, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.1@KRBTEST.COM
I20260926 22:51:25.876060 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:25.869306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:35687 (local address 127.13.148.1:42817)
0926 22:51:25.869430 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:25.869435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:25.869442 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:25.869464 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:25.869466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:25.869478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:25.869544 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:25.870048 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:25.870740 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:25.871475 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:25.871782 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:25.871840 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:25.873685 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:25.873692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:25.873693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:25.873700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:25.874958 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:25.875290 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:25.875293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:25.875294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:25.875348 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:25.875503 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:25.875504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:25.875504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:25.875637 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:25.875706 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:25.875869 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 22:51:25.875944 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260926 22:51:25.876070 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:25.869203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42817 (local address 127.13.148.2:35687)
0926 22:51:25.869252 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:25.869269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:25.869331 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:25.869560 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:25.869567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:25.869903 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:25.869973 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:25.869976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:25.870817 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:25.870818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:25.871401 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:25.871403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:25.871846 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:25.871847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:25.871920 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:25.872363 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:25.872367 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:25.873607 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:25.875030 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:25.875037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:25.875039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:25.875220 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:25.875402 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:25.875403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:25.875403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:25.875456 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:25.875699 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:25.875705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:25.875760 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 22:51:25.875832 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 22:51:25.875943 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 22:51:25.876528 28853 tablet_copy_service.cc:140] P 1e802fa25d224c5cb412a67835daacf0: Received BeginTabletCopySession request for tablet f3d2a6b74bfe4f789196c5e39f7d29cd from peer c26b5143e126414391cc065ab2288b2e ({username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:35687)
I20260926 22:51:25.876586 28853 tablet_copy_service.cc:161] P 1e802fa25d224c5cb412a67835daacf0: Beginning new tablet copy session on tablet f3d2a6b74bfe4f789196c5e39f7d29cd from peer c26b5143e126414391cc065ab2288b2e at {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:35687: session id = c26b5143e126414391cc065ab2288b2e-f3d2a6b74bfe4f789196c5e39f7d29cd
I20260926 22:51:25.877003 28853 tablet_copy_source_session.cc:205] T f3d2a6b74bfe4f789196c5e39f7d29cd P 1e802fa25d224c5cb412a67835daacf0: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 22:51:25.877897 29169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d2a6b74bfe4f789196c5e39f7d29cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:25.880307 29169 tablet_copy_client.cc:791] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: tablet copy: Starting download of 0 data blocks...
I20260926 22:51:25.880451 29169 tablet_copy_client.cc:655] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: tablet copy: Starting download of 1 WAL segments...
I20260926 22:51:25.881181 29169 tablet_copy_client.cc:531] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 22:51:25.881840 29169 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Bootstrap starting.
I20260926 22:51:25.883090 29169 log.cc:824] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:25.885267 29169 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:51:25.885450 29169 tablet_bootstrap.cc:492] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Bootstrap complete.
I20260926 22:51:25.885560 29169 ts_tablet_manager.cc:1397] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 22:51:25.886492 29169 raft_consensus.cc:348] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:25.886628 29169 raft_consensus.cc:729] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c26b5143e126414391cc065ab2288b2e, State: Initialized, Role: FOLLOWER
I20260926 22:51:25.886744 29169 consensus_queue.cc:260] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "149fd66f51e743bc802e329050394a89" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 34859 } } peers { permanent_uuid: "c26b5143e126414391cc065ab2288b2e" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 46319 } } peers { permanent_uuid: "1e802fa25d224c5cb412a67835daacf0" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42817 } }
I20260926 22:51:25.886969 29169 ts_tablet_manager.cc:1428] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260926 22:51:25.887202 28853 tablet_copy_service.cc:342] P 1e802fa25d224c5cb412a67835daacf0: Request end of tablet copy session c26b5143e126414391cc065ab2288b2e-f3d2a6b74bfe4f789196c5e39f7d29cd received from {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:35687
I20260926 22:51:25.887231 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:25.887134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:58427)
0926 22:51:25.887181 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:25.887206 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 22:51:25.887254 28853 tablet_copy_service.cc:433] P 1e802fa25d224c5cb412a67835daacf0: ending tablet copy session c26b5143e126414391cc065ab2288b2e-f3d2a6b74bfe4f789196c5e39f7d29cd on tablet f3d2a6b74bfe4f789196c5e39f7d29cd with peer c26b5143e126414391cc065ab2288b2e
I20260926 22:51:25.887526 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:25.887432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:39321)
0926 22:51:25.887478 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:25.887505 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 22:51:26.313068 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:26.312882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:37569)
0926 22:51:26.312987 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:26.313024 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":47,"threads_started":1}
I20260926 22:51:26.313109 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:26.312928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:45819)
0926 22:51:26.313035 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:26.313070 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260926 22:51:26.442566 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:26.442457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:60629)
0926 22:51:26.442501 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:26.442530 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:51:26.442582 28967 raft_consensus.cc:1203] T f3d2a6b74bfe4f789196c5e39f7d29cd P c26b5143e126414391cc065ab2288b2e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 22:51:26.443230 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:26.443063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:60385)
0926 22:51:26.443152 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:26.443184 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
I20260926 22:51:27.313705 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:27.313515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:44255)
0926 22:51:27.313626 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:27.313659 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260926 22:51:27.443317 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:27.443132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:52847)
0926 22:51:27.443242 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:27.443272 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260926 22:51:27.443759 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:27.443583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:39607)
0926 22:51:27.443692 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:27.443724 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260926 22:51:28.314355 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:28.314138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:50265)
0926 22:51:28.314270 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:28.314310 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260926 22:51:28.443928 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:28.443745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:39983)
0926 22:51:28.443849 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:28.443882 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260926 22:51:28.444301 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:28.444202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:60005)
0926 22:51:28.444252 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:28.444276 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 22:51:28.444329 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:28.444131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:49187)
0926 22:51:28.444254 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:28.444288 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260926 22:51:29.314986 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:29.314789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:50475)
0926 22:51:29.314910 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:29.314940 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260926 22:51:29.444916 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:29.444728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:47745)
0926 22:51:29.444844 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:29.444874 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 22:51:29.444964 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:29.444773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:36855)
0926 22:51:29.444894 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:29.444924 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260926 22:51:30.315644 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:30.315435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:57401)
0926 22:51:30.315562 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:30.315598 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260926 22:51:30.445611 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:30.445373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:36899)
0926 22:51:30.445529 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:30.445560 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260926 22:51:30.445611 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:30.445367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:50991)
0926 22:51:30.445521 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:30.445559 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260926 22:51:31.316285 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:31.316056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:34617)
0926 22:51:31.316204 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:31.316241 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260926 22:51:31.446230 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:31.446025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:35145)
0926 22:51:31.446160 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:31.446191 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260926 22:51:31.446223 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:31.446025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:47167)
0926 22:51:31.446143 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:31.446176 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260926 22:51:32.316891 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:32.316677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:42495)
0926 22:51:32.316807 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:32.316846 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260926 22:51:32.446817 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:32.446623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:46117)
0926 22:51:32.446739 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:32.446770 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260926 22:51:32.446841 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:32.446624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:34797)
0926 22:51:32.446766 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:32.446798 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260926 22:51:33.317523 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:33.317306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:59709)
0926 22:51:33.317444 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:33.317478 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260926 22:51:33.447422 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:33.447227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:48639)
0926 22:51:33.447358 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:33.447384 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260926 22:51:33.447422 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:33.447235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:39065)
0926 22:51:33.447348 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:33.447379 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260926 22:51:34.318272 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:34.318019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:60173)
0926 22:51:34.318176 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:34.318226 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260926 22:51:34.448136 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:34.447903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:56563)
0926 22:51:34.448038 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:34.448088 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260926 22:51:34.448136 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:34.447903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:38945)
0926 22:51:34.448059 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:34.448091 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260926 22:51:35.318930 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:35.318723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:41513)
0926 22:51:35.318842 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:35.318883 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260926 22:51:35.448771 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:35.448577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:56535)
0926 22:51:35.448700 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:35.448730 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 22:51:35.448770 29208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:35.448578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:54351)
0926 22:51:35.448696 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:35.448726 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260926 22:51:36.319626 29209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:36.319357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:55095)
0926 22:51:36.319531 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:36.319570 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
I20260926 22:51:36.449379 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:36.449192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:54579)
0926 22:51:36.449305 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:36.449336 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260926 22:51:36.449388 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:36.449218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:60071)
0926 22:51:36.449317 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:36.449346 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260926 22:51:37.320252 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:37.320031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:42327)
0926 22:51:37.320167 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:37.320205 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260926 22:51:37.450026 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:37.449792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:54835)
0926 22:51:37.449930 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:37.449959 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260926 22:51:37.450002 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:37.449792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:56295)
0926 22:51:37.449916 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:37.449954 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260926 22:51:38.320887 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:38.320685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.3:35665)
0926 22:51:38.320805 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:38.320841 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260926 22:51:38.450593 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:38.450403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.2:51445)
0926 22:51:38.450519 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:38.450550 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260926 22:51:38.450608 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:38.450434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39089 (local address 127.13.148.1:51303)
0926 22:51:38.450541 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:38.450571 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":51,"threads_started":1}
I20260926 22:51:38.670522 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 28720
I20260926 22:51:38.674849 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 28884
I20260926 22:51:38.678776 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 29016
2026-09-26T22:51:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18095 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:51:38 dist-test-slave-98hb krb5kdc[29220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:51:38 dist-test-slave-98hb krb5kdc[29220](info): set up 2 sockets
Sep 26 22:51:38 dist-test-slave-98hb krb5kdc[29220](info): commencing operation
krb5kdc: starting...
W20260926 22:51:39.633965 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:51:39 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:51:39.742085 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40447
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:39.819270 29233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:39.819456 29233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:39.819485 29233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:39.820622 29233 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 22:51:39.820658 29233 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 22:51:39.820675 29233 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 22:51:39.820689 29233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:51:39.820705 29233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:39.820730 29233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:39.820758 29233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:51:39.820776 29233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:51:39.822225 29233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:39.822446 29233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:39.822635 29233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:39.827507 29238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:39.827549 29241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:39.827528 29233 server_base.cc:1034] running on GCE node
W20260926 22:51:39.827615 29239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:39.827843 29233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:39.828190 29233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:39.829351 29233 hybrid_clock.cc:648] HybridClock initialized: now 1790463099829337 us; error 28 us; skew 500 ppm
Sep 26 22:51:39 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:39.831645 29233 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:51:39.832036 29233 webserver.cc:460] Webserver started at http://127.13.148.62:35301/ using document root <none> and password file <none>
I20260926 22:51:39.832239 29233 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:39.832284 29233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:39.832474 29233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:39.833492 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "486bdb94e9d746a29fd1a2220c5f16b9"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "7edc63c12f0d27c037eb94769b89ce12"
server_key_iv: "65d884a2944666a2221349929da0bb63"
server_key_version: "encryptionkey@0"
I20260926 22:51:39.833841 29233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "486bdb94e9d746a29fd1a2220c5f16b9"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "7edc63c12f0d27c037eb94769b89ce12"
server_key_iv: "65d884a2944666a2221349929da0bb63"
server_key_version: "encryptionkey@0"
I20260926 22:51:39.835005 29233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:51:39.835644 29248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:39.835798 29233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:51:39.835875 29233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "486bdb94e9d746a29fd1a2220c5f16b9"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "7edc63c12f0d27c037eb94769b89ce12"
server_key_iv: "65d884a2944666a2221349929da0bb63"
server_key_version: "encryptionkey@0"
I20260926 22:51:39.835940 29233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:39.852021 29233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:39.853585 29233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:39.853705 29233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:39.857223 29233 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40447
I20260926 22:51:39.857255 29300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40447 every 8 connection(s)
I20260926 22:51:39.857550 29233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:51:39.858345 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:39.860426 29301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Bootstrap starting.
I20260926 22:51:39.860955 29301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:39.861213 29301 log.cc:824] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:39.861827 29301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: No bootstrap required, opened a new log
I20260926 22:51:39.863297 29301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:51:39.863425 29301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:39.863456 29301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486bdb94e9d746a29fd1a2220c5f16b9, State: Initialized, Role: FOLLOWER
I20260926 22:51:39.863592 29301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:51:39.863662 29301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:51:39.863694 29301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:51:39.863739 29301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:39.864205 29301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:51:39.864288 29301 leader_election.cc:302] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486bdb94e9d746a29fd1a2220c5f16b9; no voters:
I20260926 22:51:39.864432 29301 leader_election.cc:288] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:51:39.864464 29304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:39.864604 29304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 1 LEADER]: Becoming Leader. State: Replica: 486bdb94e9d746a29fd1a2220c5f16b9, State: Running, Role: LEADER
I20260926 22:51:39.864658 29301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:51:39.864691 29304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:51:39.865214 29305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } } }
I20260926 22:51:39.865307 29305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:39.865221 29306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486bdb94e9d746a29fd1a2220c5f16b9. Latest consensus state: current_term: 1 leader_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } } }
I20260926 22:51:39.865402 29306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:51:39.865545 29313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:51:39.865926 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 29233
I20260926 22:51:39.866030 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:51:39.866077 29313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:51:39.866226 13904 external_mini_cluster.cc:1428] Setting key 54f649eb05270dea1dc1be5cb1a3e438
I20260926 22:51:39.867658 29313 catalog_manager.cc:1347] Generated new cluster ID: ea5d5c8ef21945c4a62fb22dd74b8613
I20260926 22:51:39.867705 29313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:51:39 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:39.875126 29313 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:51:39.875696 29313 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:51:39.876971 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:39.866771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.13.148.62:40447)
0926 22:51:39.866911 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:39.866919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:39.866948 (+ 29us) server_negotiation.cc:407] Connection header received
0926 22:51:39.867054 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:39.867061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:39.867111 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:39.867244 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:39.867876 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:39.869235 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:39.870262 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:39.870883 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:39.870991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:39.873565 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:39.873569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:39.873569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:39.873580 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:39.875420 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:39.875833 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:39.875836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:39.875836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:39.875932 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:39.876134 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:39.876135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:39.876135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:39.876387 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:39.876502 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:39.876628 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 22:51:39.876758 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260926 22:51:39.880594 29313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Generated new TSK 0
I20260926 22:51:39.880775 29313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:51:39.903848 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40447
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:39.982427 29327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:39.982621 29327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:39.982651 29327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:39.983742 29327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:39.983783 29327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:39.983816 29327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:51:39.985234 29327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:39.985457 29327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:39.985656 29327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:39.991312 29333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:39.991286 29332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:39.991298 29335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:39.991564 29327 server_base.cc:1034] running on GCE node
I20260926 22:51:39.991773 29327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:39.992033 29327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:39.993187 29327 hybrid_clock.cc:648] HybridClock initialized: now 1790463099993173 us; error 24 us; skew 500 ppm
Sep 26 22:51:39 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:39.995415 29327 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:51:39.995820 29327 webserver.cc:460] Webserver started at http://127.13.148.1:41235/ using document root <none> and password file <none>
I20260926 22:51:39.996054 29327 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:39.996104 29327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:39.996227 29327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:39.997241 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "a027e875c596404399ef056dfea1ad23"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "09958293a812eb5dc25098c0246951cd"
server_key_iv: "ffb6d86c995b1c0fc87408ac27307380"
server_key_version: "encryptionkey@0"
I20260926 22:51:39.997562 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "a027e875c596404399ef056dfea1ad23"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "09958293a812eb5dc25098c0246951cd"
server_key_iv: "ffb6d86c995b1c0fc87408ac27307380"
server_key_version: "encryptionkey@0"
I20260926 22:51:39.998845 29327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:51:39.999496 29342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:39.999637 29327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:51:39.999714 29327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "a027e875c596404399ef056dfea1ad23"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "09958293a812eb5dc25098c0246951cd"
server_key_iv: "ffb6d86c995b1c0fc87408ac27307380"
server_key_version: "encryptionkey@0"
I20260926 22:51:39.999776 29327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:40.022111 29327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:40.023823 29327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:40.023953 29327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:40.024204 29327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:40.024482 29327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:40.024520 29327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.024549 29327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:40.024572 29327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.029974 29327 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:42169
I20260926 22:51:40.030015 29455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:42169 every 8 connection(s)
I20260926 22:51:40.030308 29327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:51:40 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463099, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:40.038473 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 29327
I20260926 22:51:40.038584 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:51:40.038795 13904 external_mini_cluster.cc:1428] Setting key 23bfa8b98238c177e87ab2ea0e437be7
I20260926 22:51:40.038861 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.030968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:43061 (local address 127.13.148.62:40447)
0926 22:51:40.031012 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.031015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.031209 (+ 194us) server_negotiation.cc:407] Connection header received
0926 22:51:40.031686 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.031687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.031696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.031742 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:40.032657 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.033147 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.034194 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.034391 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.034440 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.036349 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:40.036350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:40.036351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:40.036357 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:40.037616 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:40.037971 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:40.037972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:40.037972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:40.038022 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:40.038224 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:40.038225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:40.038225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:40.038318 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:40.038364 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.038710 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.038785 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:51:40.039062 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.030952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:43061)
0926 22:51:40.031111 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.031145 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:40.031602 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:40.031806 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:40.031809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:40.032369 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:40.032560 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.032565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.033261 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.033262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.034128 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.034131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.034448 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.034448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.034546 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.034945 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:40.034953 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:40.036270 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:40.037679 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:40.037685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:40.037686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:40.037913 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:40.038083 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:40.038084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:40.038084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:40.038181 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:40.038371 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:40.038376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:40.038534 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 22:51:40.038691 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 22:51:40.038853 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260926 22:51:40.039551 29456 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40447
I20260926 22:51:40.039649 29456 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:40.039785 29456 heartbeater.cc:511] Master 127.13.148.62:40447 requested a full tablet report, sending...
I20260926 22:51:40.040256 29264 ts_manager.cc:194] Registered new tserver with Master: a027e875c596404399ef056dfea1ad23 (127.13.148.1:42169)
I20260926 22:51:40.040835 29264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:43061
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:51:40.061393 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40447
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:40.139155 29461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:40.139350 29461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:40.139371 29461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:40.140499 29461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:40.140549 29461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:40.140581 29461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:51:40.141990 29461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:40.142196 29461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:40.142429 29461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:51:40.147008 29461 server_base.cc:1034] running on GCE node
W20260926 22:51:40.146981 29467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:40.146921 29469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:40.146924 29466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:40.147321 29461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:40.147527 29461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:40.148658 29461 hybrid_clock.cc:648] HybridClock initialized: now 1790463100148645 us; error 28 us; skew 500 ppm
Sep 26 22:51:40 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:40.150866 29461 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:51:40.151229 29461 webserver.cc:460] Webserver started at http://127.13.148.2:39029/ using document root <none> and password file <none>
I20260926 22:51:40.151393 29461 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:40.151429 29461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:40.151516 29461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:40.152422 29461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "1725d337d87947f8b625813f4956f648"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "1fb8827e95ab73d056c0526299314cb8"
server_key_iv: "1a1a55a15a95854e39688b426771efcd"
server_key_version: "encryptionkey@0"
I20260926 22:51:40.152688 29461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "1725d337d87947f8b625813f4956f648"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "1fb8827e95ab73d056c0526299314cb8"
server_key_iv: "1a1a55a15a95854e39688b426771efcd"
server_key_version: "encryptionkey@0"
I20260926 22:51:40.153935 29461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:51:40.154539 29476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.154675 29461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.154740 29461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "1725d337d87947f8b625813f4956f648"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "1fb8827e95ab73d056c0526299314cb8"
server_key_iv: "1a1a55a15a95854e39688b426771efcd"
server_key_version: "encryptionkey@0"
I20260926 22:51:40.154805 29461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:40.206813 29461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:40.208949 29461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:40.209079 29461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:40.209302 29461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:40.209604 29461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:40.209637 29461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.209666 29461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:40.209682 29461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.214955 29461 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:35517
I20260926 22:51:40.215009 29589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:35517 every 8 connection(s)
I20260926 22:51:40.215303 29461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:51:40.216022 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 29461
I20260926 22:51:40.216143 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:51:40.216326 13904 external_mini_cluster.cc:1428] Setting key 3592a854bf8159fa7cea7848b31b6692
Sep 26 22:51:40 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260926 22:51:40.224050 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.215934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:46135 (local address 127.13.148.62:40447)
0926 22:51:40.215987 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.215989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.216248 (+ 259us) server_negotiation.cc:407] Connection header received
0926 22:51:40.216874 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.216875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.216883 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.216920 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:40.217904 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.218383 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.219359 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.219555 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.219609 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.221569 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:40.221571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:40.221571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:40.221579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:40.222776 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:40.223134 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:40.223135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:40.223136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:40.223188 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:40.223397 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:40.223398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:40.223398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:40.223503 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:40.223574 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.223910 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.223985 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:51:40.224213 29592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.215939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:46135)
0926 22:51:40.216125 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.216173 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:40.216768 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:40.216993 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:40.216998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:40.217583 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:40.217819 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.217826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.218466 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.218467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.219279 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.219282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.219622 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.219623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.219752 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.220212 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:40.220220 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:40.221480 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:40.222848 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:40.222855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:40.222857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:40.223069 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:40.223241 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:40.223242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:40.223242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:40.223344 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:40.223581 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:40.223587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:40.223749 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 22:51:40.223859 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 22:51:40.224018 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260926 22:51:40.224713 29590 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40447
I20260926 22:51:40.224792 29590 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:40.224925 29590 heartbeater.cc:511] Master 127.13.148.62:40447 requested a full tablet report, sending...
I20260926 22:51:40.225244 29264 ts_manager.cc:194] Registered new tserver with Master: 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
I20260926 22:51:40.225595 29264 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:46135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:51:40.237633 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40447
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:51:40.314723 29595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:51:40.314908 29595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:51:40.314937 29595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:51:40.316041 29595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:51:40.316078 29595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:51:40.316111 29595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:51:40.317456 29595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:51:40.317678 29595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:51:40.317893 29595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:51:40.322422 29603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:40.322392 29601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:51:40.322398 29600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:51:40.322703 29595 server_base.cc:1034] running on GCE node
I20260926 22:51:40.322882 29595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:51:40.323091 29595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:51:40.324218 29595 hybrid_clock.cc:648] HybridClock initialized: now 1790463100324205 us; error 29 us; skew 500 ppm
Sep 26 22:51:40 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:51:40.326479 29595 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:51:40.326893 29595 webserver.cc:460] Webserver started at http://127.13.148.3:43341/ using document root <none> and password file <none>
I20260926 22:51:40.327071 29595 fs_manager.cc:359] Metadata directory not provided
I20260926 22:51:40.327116 29595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:51:40.327234 29595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:51:40.328157 29595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "ab1dd394854b203e718053824ff8b004"
server_key_iv: "28c7a856dd03a515eda08eaaad51657b"
server_key_version: "encryptionkey@0"
I20260926 22:51:40.328459 29595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "ab1dd394854b203e718053824ff8b004"
server_key_iv: "28c7a856dd03a515eda08eaaad51657b"
server_key_version: "encryptionkey@0"
I20260926 22:51:40.329589 29595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:51:40.330276 29610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.330422 29595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.330497 29595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "ab1dd394854b203e718053824ff8b004"
server_key_iv: "28c7a856dd03a515eda08eaaad51657b"
server_key_version: "encryptionkey@0"
I20260926 22:51:40.330557 29595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:51:40.354807 29595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:51:40.356817 29595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:51:40.356947 29595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:51:40.357158 29595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:51:40.357460 29595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:51:40.357491 29595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.357518 29595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:51:40.357535 29595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:51:40.363088 29595 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:46743
I20260926 22:51:40.363139 29723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:46743 every 8 connection(s)
I20260926 22:51:40.363410 29595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
Sep 26 22:51:40 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463100, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:51:40.371454 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.364028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:56715 (local address 127.13.148.62:40447)
0926 22:51:40.364084 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.364086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.364240 (+ 154us) server_negotiation.cc:407] Connection header received
0926 22:51:40.364724 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.364726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.364734 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.364796 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:40.365820 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.366354 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.367315 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.367495 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.367541 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.369329 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:40.369330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:40.369330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:40.369336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:40.370405 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:40.370727 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:40.370728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:40.370728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:40.370771 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:40.371006 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:40.371006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:40.371007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:40.371094 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:40.371134 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.371341 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.371393 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:51:40.371641 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.363991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:56715)
0926 22:51:40.364131 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.364167 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:40.364651 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:40.364884 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:40.364887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:40.365488 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:51:40.365716 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.365721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.366438 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.366440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.367249 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.367251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.367562 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.367563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.367654 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.368014 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:51:40.368021 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:51:40.369262 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:51:40.370465 (+ 1203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:40.370470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:40.370471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:40.370671 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:40.370842 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:51:40.370843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:51:40.370843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:51:40.370949 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:51:40.371137 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:51:40.371140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:51:40.371246 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:51:40.371324 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:51:40.371438 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260926 22:51:40.371814 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 29595
I20260926 22:51:40.371920 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:51:40.372069 29724 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40447
I20260926 22:51:40.372112 13904 external_mini_cluster.cc:1428] Setting key 8137f9beaf610a145baa79a865d29a2e
I20260926 22:51:40.372148 29724 heartbeater.cc:464] Registering TS with master...
I20260926 22:51:40.372304 29724 heartbeater.cc:511] Master 127.13.148.62:40447 requested a full tablet report, sending...
I20260926 22:51:40.372663 29265 ts_manager.cc:194] Registered new tserver with Master: 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743)
I20260926 22:51:40.373013 29265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:56715
I20260926 22:51:40.383503 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:51:40.389618 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.384701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56416 (local address 127.13.148.62:40447)
0926 22:51:40.384744 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.384746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.384833 (+ 87us) server_negotiation.cc:407] Connection header received
0926 22:51:40.384885 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.384885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.384893 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.384928 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:51:40.385390 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.385894 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.386556 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.386730 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.386770 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.387802 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:51:40.387804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:51:40.387804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:51:40.387810 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:51:40.388808 (+ 998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:40.389074 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:40.389075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:40.389075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:40.389117 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:51:40.389239 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:51:40.389239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:51:40.389239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:51:40.389353 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:51:40.389422 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.389491 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.389559 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:51:40.390866 29265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:51:40.392316 29265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:51:40.398588 29737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.394268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42169 (local address 127.0.0.1:54110)
0926 22:51:40.394627 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.394635 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:40.394682 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:40.394873 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:40.394876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:40.394878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:40.394944 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.394946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.396214 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.396215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.397432 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.397435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.398279 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.398280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.398367 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.398383 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 22:51:40.398449 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 22:51:40.398499 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":8,"thread_start_us":45,"threads_started":1}
I20260926 22:51:40.398590 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.394443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.13.148.1:42169)
0926 22:51:40.394598 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.394602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.394676 (+ 74us) server_negotiation.cc:407] Connection header received
0926 22:51:40.394745 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.394747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.394763 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.394841 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:40.395011 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.396119 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.397505 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.398199 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.398271 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.398326 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.398471 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.398512 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260926 22:51:40.398873 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.394274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35517 (local address 127.0.0.1:36594)
0926 22:51:40.394763 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.394773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:40.394832 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:40.395230 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:40.395230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:40.395232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:40.395268 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.395270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.396106 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.396107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.397432 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.397435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.398592 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.398593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.398710 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.398716 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 22:51:40.398766 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 22:51:40.398808 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":32,"thread_start_us":86,"threads_started":1}
I20260926 22:51:40.398902 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.394518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.13.148.2:35517)
0926 22:51:40.394704 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.394708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.395016 (+ 308us) server_negotiation.cc:407] Connection header received
0926 22:51:40.395068 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.395070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.395085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.395150 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:40.395321 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.396013 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.397786 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.398447 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.398509 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.398559 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.398790 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.398832 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260926 22:51:40.399598 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.394268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46743 (local address 127.0.0.1:39072)
0926 22:51:40.394411 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.394446 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:40.394511 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:40.394912 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:40.394913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:40.394915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:40.394964 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.394965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.397535 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.397536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.398254 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.398256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.399402 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.399403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.399467 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.399472 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:51:40.399496 (+ 24us) client_negotiation.cc:240] Negotiation successful
0926 22:51:40.399543 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260926 22:51:40.399504 29390 tablet_service.cc:1467] Processing CreateTablet for tablet d5a1f4d7e9474eeab3af7a0ea8a25072 (DEFAULT_TABLE table=test-table [id=f08970b21a2844849355dfb5feb978a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:40.399616 29524 tablet_service.cc:1467] Processing CreateTablet for tablet d5a1f4d7e9474eeab3af7a0ea8a25072 (DEFAULT_TABLE table=test-table [id=f08970b21a2844849355dfb5feb978a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:40.399842 29390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a1f4d7e9474eeab3af7a0ea8a25072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:40.399879 29524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a1f4d7e9474eeab3af7a0ea8a25072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:40.399899 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.394613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.13.148.3:46743)
0926 22:51:40.394788 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.394791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.394798 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:40.394829 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.394831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.394846 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.394916 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:40.395039 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.396127 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.398351 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.399311 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.399699 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.399766 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.399791 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.399829 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260926 22:51:40.403044 29742 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Bootstrap starting.
I20260926 22:51:40.403218 29741 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Bootstrap starting.
I20260926 22:51:40.403611 29658 tablet_service.cc:1467] Processing CreateTablet for tablet d5a1f4d7e9474eeab3af7a0ea8a25072 (DEFAULT_TABLE table=test-table [id=f08970b21a2844849355dfb5feb978a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:51:40.403743 29742 tablet_bootstrap.cc:654] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:40.403885 29658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a1f4d7e9474eeab3af7a0ea8a25072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:51:40.403903 29741 tablet_bootstrap.cc:654] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:40.404294 29742 log.cc:824] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:40.404460 29741 log.cc:824] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:40.405038 29742 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: No bootstrap required, opened a new log
I20260926 22:51:40.405108 29742 ts_tablet_manager.cc:1397] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:51:40.405159 29741 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: No bootstrap required, opened a new log
I20260926 22:51:40.405220 29741 ts_tablet_manager.cc:1397] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:40.406635 29745 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Bootstrap starting.
I20260926 22:51:40.407142 29745 tablet_bootstrap.cc:654] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Neither blocks nor log segments found. Creating new log.
I20260926 22:51:40.407039 29741 raft_consensus.cc:348] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.407003 29742 raft_consensus.cc:348] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.407187 29741 raft_consensus.cc:374] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:40.407187 29742 raft_consensus.cc:374] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:40.407223 29742 raft_consensus.cc:729] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1725d337d87947f8b625813f4956f648, State: Initialized, Role: FOLLOWER
I20260926 22:51:40.407223 29741 raft_consensus.cc:729] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a027e875c596404399ef056dfea1ad23, State: Initialized, Role: FOLLOWER
I20260926 22:51:40.407344 29742 consensus_queue.cc:260] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.407351 29741 consensus_queue.cc:260] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.407609 29741 ts_tablet_manager.cc:1428] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:51:40.407609 29742 ts_tablet_manager.cc:1428] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:51:40.407641 29745 log.cc:824] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:51:40.407727 29456 heartbeater.cc:503] Master 127.13.148.62:40447 was elected leader, sending a full tablet report...
I20260926 22:51:40.407737 29590 heartbeater.cc:503] Master 127.13.148.62:40447 was elected leader, sending a full tablet report...
I20260926 22:51:40.408339 29745 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: No bootstrap required, opened a new log
I20260926 22:51:40.408403 29745 ts_tablet_manager.cc:1397] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:51:40.410018 29745 raft_consensus.cc:348] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.410146 29745 raft_consensus.cc:374] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:51:40.410178 29745 raft_consensus.cc:729] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd4ab03f5af4c8184046fa0e3dc2c9d, State: Initialized, Role: FOLLOWER
I20260926 22:51:40.410282 29745 consensus_queue.cc:260] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.410494 29745 ts_tablet_manager.cc:1428] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:51:40.410526 29724 heartbeater.cc:503] Master 127.13.148.62:40447 was elected leader, sending a full tablet report...
W20260926 22:51:40.465791 29591 tablet.cc:2369] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:51:40.495703 29747 raft_consensus.cc:482] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:51:40.495817 29747 raft_consensus.cc:504] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.496202 29747 leader_election.cc:288] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743), 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
I20260926 22:51:40.498865 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.496459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:41597 (local address 127.13.148.3:46743)
0926 22:51:40.496501 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.496503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.496510 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:51:40.496547 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.496548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.496555 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.496597 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:40.496753 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.497217 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.498186 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.498560 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.498621 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.498630 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.498767 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.498802 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:51:40.498865 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.496351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46743 (local address 127.13.148.1:41597)
0926 22:51:40.496398 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.496408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:40.496445 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:40.496617 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:40.496618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:40.496620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:40.496691 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.496693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.497293 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.497294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.498104 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.498106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.498641 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.498642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.498709 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.498712 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:51:40.498756 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:51:40.498797 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 22:51:40.498982 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.496434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:51379 (local address 127.13.148.2:35517)
0926 22:51:40.496476 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.496479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.496639 (+ 160us) server_negotiation.cc:407] Connection header received
0926 22:51:40.496693 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.496694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.496704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.496770 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:51:40.496906 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.497408 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.498310 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.498679 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.498740 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.498748 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.498875 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.498911 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:51:40.499126 29750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.496351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35517 (local address 127.13.148.1:51379)
0926 22:51:40.496571 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.496580 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:51:40.496638 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:51:40.496777 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:51:40.496778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:51:40.496780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:51:40.496851 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.496852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.497485 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.497485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.498230 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:51:40.498232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.498750 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.498751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:51:40.498818 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.498821 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:51:40.498865 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:51:40.498902 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":58,"thread_start_us":108,"threads_started":1}
I20260926 22:51:40.499212 29678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a1f4d7e9474eeab3af7a0ea8a25072" candidate_uuid: "a027e875c596404399ef056dfea1ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" is_pre_election: true
I20260926 22:51:40.499372 29678 raft_consensus.cc:2454] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a027e875c596404399ef056dfea1ad23 in term 0.
I20260926 22:51:40.499464 29544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a1f4d7e9474eeab3af7a0ea8a25072" candidate_uuid: "a027e875c596404399ef056dfea1ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1725d337d87947f8b625813f4956f648" is_pre_election: true
I20260926 22:51:40.499609 29544 raft_consensus.cc:2454] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a027e875c596404399ef056dfea1ad23 in term 0.
I20260926 22:51:40.499601 29343 leader_election.cc:302] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd4ab03f5af4c8184046fa0e3dc2c9d, a027e875c596404399ef056dfea1ad23; no voters:
I20260926 22:51:40.499732 29747 raft_consensus.cc:2781] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:51:40.499796 29747 raft_consensus.cc:482] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:51:40.499825 29747 raft_consensus.cc:3037] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:40.500491 29747 raft_consensus.cc:504] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.500604 29747 leader_election.cc:288] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743), 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
I20260926 22:51:40.500756 29678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a1f4d7e9474eeab3af7a0ea8a25072" candidate_uuid: "a027e875c596404399ef056dfea1ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d"
I20260926 22:51:40.500768 29544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a1f4d7e9474eeab3af7a0ea8a25072" candidate_uuid: "a027e875c596404399ef056dfea1ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1725d337d87947f8b625813f4956f648"
I20260926 22:51:40.500823 29678 raft_consensus.cc:3037] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:40.500830 29544 raft_consensus.cc:3037] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:51:40.501520 29544 raft_consensus.cc:2454] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a027e875c596404399ef056dfea1ad23 in term 1.
I20260926 22:51:40.501519 29678 raft_consensus.cc:2454] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a027e875c596404399ef056dfea1ad23 in term 1.
I20260926 22:51:40.501703 29346 leader_election.cc:302] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1725d337d87947f8b625813f4956f648, a027e875c596404399ef056dfea1ad23; no voters:
I20260926 22:51:40.501818 29747 raft_consensus.cc:2781] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:51:40.501935 29747 raft_consensus.cc:686] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 1 LEADER]: Becoming Leader. State: Replica: a027e875c596404399ef056dfea1ad23, State: Running, Role: LEADER
I20260926 22:51:40.502018 29747 consensus_queue.cc:237] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:51:40.502825 29265 catalog_manager.cc:5495] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 reported cstate change: term changed from 0 to 1, leader changed from <none> to a027e875c596404399ef056dfea1ad23 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "a027e875c596404399ef056dfea1ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } health_report { overall_health: UNKNOWN } } }
I20260926 22:51:40.514889 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.512941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.13.148.1:42169)
0926 22:51:40.512979 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:51:40.512980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:51:40.512988 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:51:40.513015 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:51:40.513015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:51:40.513030 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:51:40.513082 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:51:40.513167 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.513608 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.514316 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:51:40.514493 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:51:40.514533 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:51:40.514638 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:51:40.514730 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:51:40.514784 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:51:40.514795 (+ 11us) server_negotiation.cc:299] Negotiation successful
0926 22:51:40.514809 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:51:40.516448 29678 raft_consensus.cc:1261] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 1 FOLLOWER]: Refusing update from remote peer a027e875c596404399ef056dfea1ad23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:40.516453 29544 raft_consensus.cc:1261] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 1 FOLLOWER]: Refusing update from remote peer a027e875c596404399ef056dfea1ad23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:51:40.516749 29751 consensus_queue.cc:1035] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:40.516861 29747 consensus_queue.cc:1035] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:51:40.518989 29754 mvcc.cc:205] Tried to move back new op lower bound from 7333736859713073152 to 7333736859656765440. Current Snapshot: MvccSnapshot[applied={T|T < 7333736859713073152}]
I20260926 22:51:40.519402 29755 mvcc.cc:205] Tried to move back new op lower bound from 7333736859713073152 to 7333736859656765440. Current Snapshot: MvccSnapshot[applied={T|T < 7333736859713073152}]
I20260926 22:51:40.519729 29756 mvcc.cc:205] Tried to move back new op lower bound from 7333736859713073152 to 7333736859656765440. Current Snapshot: MvccSnapshot[applied={T|T < 7333736859713073152 or (T in {7333736859713073152})}]
I20260926 22:51:40.521588 29264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:51:40.521670 29264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:51:40.522346 29264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Sending DeleteTablet for 3 replicas of tablet d5a1f4d7e9474eeab3af7a0ea8a25072
I20260926 22:51:40.522650 29658 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a1f4d7e9474eeab3af7a0ea8a25072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:40 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:39072
I20260926 22:51:40.522715 29524 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a1f4d7e9474eeab3af7a0ea8a25072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:40 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36594
I20260926 22:51:40.522701 29390 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a1f4d7e9474eeab3af7a0ea8a25072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:40 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:54110
I20260926 22:51:40.522902 29763 tablet_replica.cc:330] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: stopping tablet replica
I20260926 22:51:40.522910 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 29327
I20260926 22:51:40.523000 29763 raft_consensus.cc:2229] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:51:40.523089 29763 raft_consensus.cc:2258] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:40.523985 29764 tablet_replica.cc:330] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: stopping tablet replica
I20260926 22:51:40.524094 29764 raft_consensus.cc:2229] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:51:40.524195 29764 raft_consensus.cc:2258] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:51:40.524345 29763 ts_tablet_manager.cc:1905] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:40.524561 29764 ts_tablet_manager.cc:1905] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:51:40.526000 29763 ts_tablet_manager.cc:1918] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:51:40.526062 29763 log.cc:1196] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/d5a1f4d7e9474eeab3af7a0ea8a25072
I20260926 22:51:40.526111 29764 ts_tablet_manager.cc:1918] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:51:40.526170 29764 log.cc:1196] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/d5a1f4d7e9474eeab3af7a0ea8a25072
I20260926 22:51:40.526319 29763 ts_tablet_manager.cc:1939] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Deleting consensus metadata
I20260926 22:51:40.526391 29764 ts_tablet_manager.cc:1939] T d5a1f4d7e9474eeab3af7a0ea8a25072 P 1725d337d87947f8b625813f4956f648: Deleting consensus metadata
I20260926 22:51:40.526705 29249 catalog_manager.cc:4841] TS 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743): tablet d5a1f4d7e9474eeab3af7a0ea8a25072 (table test-table [id=f08970b21a2844849355dfb5feb978a9]) successfully deleted
I20260926 22:51:40.526767 29252 catalog_manager.cc:4841] TS 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517): tablet d5a1f4d7e9474eeab3af7a0ea8a25072 (table test-table [id=f08970b21a2844849355dfb5feb978a9]) successfully deleted
W20260926 22:51:40.527837 29251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:42169 (error 108)
I20260926 22:51:40.528049 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 29461
I20260926 22:51:40.528640 29735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:51:40.528164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42169 (local address 127.0.0.1:54128)
0926 22:51:40.528569 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 22:51:40.528608 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
W20260926 22:51:40.528887 29251 catalog_manager.cc:4584] TS a027e875c596404399ef056dfea1ad23 (127.13.148.1:42169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5a1f4d7e9474eeab3af7a0ea8a25072: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42169: connect: Connection refused (error 111)
I20260926 22:51:40.532588 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 29595
I20260926 22:51:40.536198 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 29233
I20260926 22:52:00.540447 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40447
--webserver_interface=127.13.148.62
--webserver_port=35301
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:00.618468 29767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:00.618681 29767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:00.618711 29767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:00.619836 29767 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 22:52:00.619875 29767 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 22:52:00.619890 29767 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 22:52:00.619904 29767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:00.619921 29767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:00.619937 29767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:00.619954 29767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:00.619976 29767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:00.621418 29767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=35301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:00.621667 29767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:00.621897 29767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:00.630393 29773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:00.630419 29767 server_base.cc:1034] running on GCE node
W20260926 22:52:00.630398 29776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:00.630393 29774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:00.630942 29767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:00.631217 29767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:00.632341 29767 hybrid_clock.cc:648] HybridClock initialized: now 1790463120632334 us; error 27 us; skew 500 ppm
Sep 26 22:52:00 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:00.638037 29767 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:52:00.638473 29767 webserver.cc:460] Webserver started at http://127.13.148.62:35301/ using document root <none> and password file <none>
I20260926 22:52:00.638696 29767 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:00.638754 29767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:00.640131 29767 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:00.640812 29783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:00.640976 29767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:52:00.641067 29767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "486bdb94e9d746a29fd1a2220c5f16b9"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "7edc63c12f0d27c037eb94769b89ce12"
server_key_iv: "65d884a2944666a2221349929da0bb63"
server_key_version: "encryptionkey@0"
I20260926 22:52:00.641513 29767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:00.652639 29767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:00.654599 29767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:00.654731 29767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:00.658350 29767 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40447
I20260926 22:52:00.658385 29835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40447 every 8 connection(s)
I20260926 22:52:00.658738 29767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:00.660072 29836 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:52:00.660696 29836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Bootstrap starting.
I20260926 22:52:00.665184 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 29767
I20260926 22:52:00.665498 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:42169
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:40447
--webserver_port=41235
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:52:00.669341 29836 log.cc:824] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:00.671757 29836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:52:00.672080 29836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Bootstrap complete.
I20260926 22:52:00.675242 29836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:52:00.675557 29836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486bdb94e9d746a29fd1a2220c5f16b9, State: Initialized, Role: FOLLOWER
I20260926 22:52:00.675733 29836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:52:00.675810 29836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:00.675864 29836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:00.675915 29836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 22:52:00.677205 29836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:52:00.677309 29836 leader_election.cc:302] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486bdb94e9d746a29fd1a2220c5f16b9; no voters:
I20260926 22:52:00.677451 29836 leader_election.cc:288] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 22:52:00.677505 29840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 22:52:00.677656 29840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 2 LEADER]: Becoming Leader. State: Replica: 486bdb94e9d746a29fd1a2220c5f16b9, State: Running, Role: LEADER
I20260926 22:52:00.677685 29836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:00.677767 29840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:52:00.678056 29841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } } }
I20260926 22:52:00.678136 29841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:00.678068 29842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486bdb94e9d746a29fd1a2220c5f16b9. Latest consensus state: current_term: 2 leader_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } } }
I20260926 22:52:00.678246 29842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:00.678403 29848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:00.679342 29848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:00.679462 29848 catalog_manager.cc:1259] Loaded cluster ID: ea5d5c8ef21945c4a62fb22dd74b8613
I20260926 22:52:00.679498 29848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:00.680387 29848 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:00.680544 29848 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Loaded TSK: 0
I20260926 22:52:00.687207 29848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Generated new TSK 1
I20260926 22:52:00.687282 29848 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:52:00.755424 29838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:00.755626 29838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:00.755657 29838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:00.756795 29838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:00.756837 29838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:00.756870 29838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:52:00.758294 29838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:42169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=41235
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:00.758510 29838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:00.758716 29838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:00.763530 29864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:00.763552 29863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:00.763701 29838 server_base.cc:1034] running on GCE node
W20260926 22:52:00.763559 29866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:00.764041 29838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:00.764278 29838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:00.765413 29838 hybrid_clock.cc:648] HybridClock initialized: now 1790463120765401 us; error 30 us; skew 500 ppm
Sep 26 22:52:00 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:00.768011 29838 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:52:00.768556 29838 webserver.cc:460] Webserver started at http://127.13.148.1:41235/ using document root <none> and password file <none>
I20260926 22:52:00.768806 29838 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:00.768886 29838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:00.770568 29838 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:00.771322 29873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:00.771516 29838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:00.771608 29838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "a027e875c596404399ef056dfea1ad23"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "09958293a812eb5dc25098c0246951cd"
server_key_iv: "ffb6d86c995b1c0fc87408ac27307380"
server_key_version: "encryptionkey@0"
I20260926 22:52:00.772033 29838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:00.787422 29838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:00.789886 29838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:00.790055 29838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:00.790318 29838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:00.790814 29880 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:52:00.792353 29838 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:52:00.792412 29838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 22:52:00.792446 29838 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:52:00.793324 29838 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:52:00.793371 29838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 22:52:00.793443 29880 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Bootstrap starting.
I20260926 22:52:00.800462 29838 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:42169
I20260926 22:52:00.800719 29987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:42169 every 8 connection(s)
I20260926 22:52:00.800932 29838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:52:00.801177 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 29838
I20260926 22:52:00.801491 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:35517
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:40447
--webserver_port=39029
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:52:00.804036 29880 log.cc:824] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:00.806625 29880 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:52:00.806921 29880 tablet_bootstrap.cc:492] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Bootstrap complete.
I20260926 22:52:00.808023 29880 ts_tablet_manager.cc:1397] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260926 22:52:00.809442 29880 raft_consensus.cc:348] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:52:00.809886 29880 raft_consensus.cc:729] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a027e875c596404399ef056dfea1ad23, State: Initialized, Role: FOLLOWER
I20260926 22:52:00.810046 29880 consensus_queue.cc:260] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } } peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } }
I20260926 22:52:00.810319 29880 ts_tablet_manager.cc:1428] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 26 22:52:00 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:00.817299 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:00.802354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:55715)
0926 22:52:00.804214 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:00.804260 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:00.804821 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:00.806449 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:00.806453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:00.806800 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:00.807075 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:00.807081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.808840 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.808841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:00.810125 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:00.810128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.810899 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.810901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:00.811045 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:00.811680 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:00.811690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:00.813482 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:00.815561 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:00.815569 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:00.815572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:00.815870 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:00.816135 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:00.816136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:00.816136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:00.816261 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:00.816670 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:00.816676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:00.816824 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 22:52:00.816927 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 22:52:00.817092 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1809,"thread_start_us":89,"threads_started":1}
I20260926 22:52:00.817302 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:00.802445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:55715 (local address 127.13.148.62:40447)
0926 22:52:00.805868 (+ 3423us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:00.805878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:00.805924 (+ 46us) server_negotiation.cc:407] Connection header received
0926 22:52:00.806099 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:00.806106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:00.806163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:00.806311 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:00.807209 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.808720 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.810220 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.810797 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.810902 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:00.813599 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:00.813604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:00.813604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:00.813618 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:00.815476 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:00.815947 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:00.815950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:00.815950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:00.816073 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:00.816316 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:00.816317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:00.816317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:00.816608 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:00.816683 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:00.816981 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 22:52:00.817114 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3369,"thread_start_us":89,"threads_started":1}
I20260926 22:52:00.818612 29988 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40447
I20260926 22:52:00.818715 29988 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:00.818882 29988 heartbeater.cc:511] Master 127.13.148.62:40447 requested a full tablet report, sending...
I20260926 22:52:00.819633 29800 ts_manager.cc:194] Registered new tserver with Master: a027e875c596404399ef056dfea1ad23 (127.13.148.1:42169)
I20260926 22:52:00.820822 29800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:55715
I20260926 22:52:00.821832 29988 heartbeater.cc:503] Master 127.13.148.62:40447 was elected leader, sending a full tablet report...
I20260926 22:52:00.825340 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:00.820290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42169 (local address 127.0.0.1:53034)
0926 22:52:00.821714 (+ 1424us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:00.821770 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:00.821849 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:00.822030 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:00.822033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:00.822035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:00.822096 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:00.822097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.823388 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.823388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:00.824181 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:00.824184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.825102 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.825103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:00.825171 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:00.825181 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 22:52:00.825245 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 22:52:00.825286 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1398,"thread_start_us":104,"threads_started":1}
I20260926 22:52:00.825369 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:00.821105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.13.148.1:42169)
0926 22:52:00.821267 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:00.821271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:00.821836 (+ 565us) server_negotiation.cc:407] Connection header received
0926 22:52:00.821924 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:00.821926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:00.821941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:00.822018 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:00.822222 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.823280 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.824257 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.824959 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.825026 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:00.825078 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:00.825269 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 22:52:00.825307 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260926 22:52:00.825687 29922 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a1f4d7e9474eeab3af7a0ea8a25072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:51:40 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:53034
I20260926 22:52:00.826292 29997 tablet_replica.cc:330] stopping tablet replica
I20260926 22:52:00.826408 29997 raft_consensus.cc:2229] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:00.826478 29997 raft_consensus.cc:2258] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:00.826789 29997 ts_tablet_manager.cc:1905] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:00.828884 29997 ts_tablet_manager.cc:1918] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:00.828945 29997 log.cc:1196] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal/wals/d5a1f4d7e9474eeab3af7a0ea8a25072
I20260926 22:52:00.829162 29997 ts_tablet_manager.cc:1939] T d5a1f4d7e9474eeab3af7a0ea8a25072 P a027e875c596404399ef056dfea1ad23: Deleting consensus metadata
I20260926 22:52:00.829612 29786 catalog_manager.cc:4841] TS a027e875c596404399ef056dfea1ad23 (127.13.148.1:42169): tablet d5a1f4d7e9474eeab3af7a0ea8a25072 (table test-table [id=f08970b21a2844849355dfb5feb978a9]) successfully deleted
W20260926 22:52:00.888263 29992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:00.888458 29992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:00.888489 29992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:00.889581 29992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:00.889619 29992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:00.889654 29992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:52:00.891050 29992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:35517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=39029
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:00.891265 29992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:00.891464 29992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:00.896049 30002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:00.896056 30005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:00.896095 30003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:00.896270 29992 server_base.cc:1034] running on GCE node
I20260926 22:52:00.896451 29992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:00.896690 29992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:00.897789 29992 hybrid_clock.cc:648] HybridClock initialized: now 1790463120897754 us; error 48 us; skew 500 ppm
Sep 26 22:52:00 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:00.900125 29992 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:52:00.900513 29992 webserver.cc:460] Webserver started at http://127.13.148.2:39029/ using document root <none> and password file <none>
I20260926 22:52:00.900691 29992 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:00.900740 29992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:00.902061 29992 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:00.902603 30012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:00.902770 29992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:00.902848 29992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "1725d337d87947f8b625813f4956f648"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "1fb8827e95ab73d056c0526299314cb8"
server_key_iv: "1a1a55a15a95854e39688b426771efcd"
server_key_version: "encryptionkey@0"
I20260926 22:52:00.903158 29992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:00.920121 29992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:00.921845 29992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:00.921963 29992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:00.922178 29992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:00.922475 29992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:00.922506 29992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:00.922534 29992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:00.922551 29992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:00.927702 29992 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:35517
I20260926 22:52:00.927765 30125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:35517 every 8 connection(s)
I20260926 22:52:00.928061 29992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:52:00 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463120, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:00.936049 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:00.928721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:35333 (local address 127.13.148.62:40447)
0926 22:52:00.928776 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:00.928779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:00.928979 (+ 200us) server_negotiation.cc:407] Connection header received
0926 22:52:00.929468 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:00.929469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:00.929482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:00.929549 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:00.930488 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.930993 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.931913 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.932112 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.932167 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:00.934019 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:00.934020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:00.934020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:00.934027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:00.935062 (+ 1035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:00.935375 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:00.935375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:00.935376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:00.935418 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:00.935565 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:00.935566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:00.935566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:00.935668 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:00.935712 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:00.935909 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 22:52:00.935973 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:52:00.936220 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:00.928706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:35333)
0926 22:52:00.928866 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:00.928907 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:00.929359 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:00.929609 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:00.929613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:00.930209 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:00.930408 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:00.930412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.931060 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.931061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:00.931839 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:00.931841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:00.932178 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:00.932179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:00.932271 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:00.932668 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:00.932675 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:00.933947 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:00.935122 (+ 1175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:00.935128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:00.935130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:00.935322 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:00.935453 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:00.935453 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:00.935453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:00.935529 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:00.935710 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:00.935713 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:00.935818 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 22:52:00.935893 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 22:52:00.936016 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260926 22:52:00.936664 30126 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40447
I20260926 22:52:00.936748 30126 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:00.936908 30126 heartbeater.cc:511] Master 127.13.148.62:40447 requested a full tablet report, sending...
I20260926 22:52:00.937243 29800 ts_manager.cc:194] Registered new tserver with Master: 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
I20260926 22:52:00.937565 29800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:35333
I20260926 22:52:00.938210 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 29992
I20260926 22:52:00.938441 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:46743
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:40447
--webserver_port=43341
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:01.016577 30129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:01.016780 30129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:01.016803 30129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:01.017902 30129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:01.017937 30129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:01.017961 30129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:52:01.019316 30129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:46743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=43341
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:01.019501 30129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:01.019676 30129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:01.024315 30137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:01.024315 30135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:01.024324 30134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:01.024520 30129 server_base.cc:1034] running on GCE node
I20260926 22:52:01.024717 30129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:01.024938 30129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:01.026077 30129 hybrid_clock.cc:648] HybridClock initialized: now 1790463121026057 us; error 33 us; skew 500 ppm
Sep 26 22:52:01 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463121, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:01.028347 30129 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:52:01.028725 30129 webserver.cc:460] Webserver started at http://127.13.148.3:43341/ using document root <none> and password file <none>
I20260926 22:52:01.028914 30129 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:01.028971 30129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:01.030345 30129 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:01.030917 30144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:01.031078 30129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:01.031158 30129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "ab1dd394854b203e718053824ff8b004"
server_key_iv: "28c7a856dd03a515eda08eaaad51657b"
server_key_version: "encryptionkey@0"
I20260926 22:52:01.031473 30129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:01.039875 30129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:01.041746 30129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:01.041874 30129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:01.042109 30129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:01.042382 30129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:01.042413 30129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:01.042435 30129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:01.042457 30129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:01.047806 30129 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:46743
I20260926 22:52:01.047864 30257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:46743 every 8 connection(s)
I20260926 22:52:01.048184 30129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:52:01.052807 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 30129
Sep 26 22:52:01 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463121, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:01.057439 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.048966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:36265 (local address 127.13.148.62:40447)
0926 22:52:01.049037 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.049039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.049208 (+ 169us) server_negotiation.cc:407] Connection header received
0926 22:52:01.049667 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.049668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.049677 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.049767 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:01.050707 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.051219 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.052269 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.052474 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.052526 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.054767 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:01.054769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:01.054769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:01.054777 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:01.056117 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:01.056567 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:01.056568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:01.056569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:01.056639 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:01.056867 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:01.056868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:01.056869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:01.056998 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:01.057065 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:01.057291 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 22:52:01.057372 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:52:01.057641 30260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.048938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:36265)
0926 22:52:01.049086 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:01.049128 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:01.049595 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:01.049825 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:01.049828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:01.050414 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:01.050616 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.050622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.051342 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.051343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.052173 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.052175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.052571 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.052572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.052686 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.053198 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:01.053210 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:01.054680 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:01.056202 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:01.056208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:01.056210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:01.056498 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:01.056703 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:01.056703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:01.056704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:01.056807 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:01.057054 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:01.057057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:01.057171 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 22:52:01.057249 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 22:52:01.057400 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260926 22:52:01.058107 30258 heartbeater.cc:347] Connected to a master server at 127.13.148.62:40447
I20260926 22:52:01.058211 30258 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:01.058403 30258 heartbeater.cc:511] Master 127.13.148.62:40447 requested a full tablet report, sending...
I20260926 22:52:01.058755 29800 ts_manager.cc:194] Registered new tserver with Master: 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743)
I20260926 22:52:01.059098 29800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:36265
I20260926 22:52:01.059791 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.053218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.13.148.62:40447)
0926 22:52:01.053321 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.053323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.053413 (+ 90us) server_negotiation.cc:407] Connection header received
0926 22:52:01.053469 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.053470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.053480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.053536 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:01.054261 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.054762 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.055771 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.055971 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.056079 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.057338 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:01.057339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:01.057340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:01.057346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:01.058796 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:01.059115 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:01.059116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:01.059116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:01.059171 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:01.059360 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:01.059361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:01.059361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:01.059469 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:01.059582 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:01.059671 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 22:52:01.059729 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
I20260926 22:52:01.060451 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:52:01.063102 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.060739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.13.148.62:40447)
0926 22:52:01.060771 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.060772 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.060896 (+ 124us) server_negotiation.cc:407] Connection header received
0926 22:52:01.060963 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.060964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.061001 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.061059 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:52:01.061168 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.061606 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.062356 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.062553 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.062605 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.062755 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:52:01.062794 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 22:52:01.063027 (+ 233us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54316: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:52:01.063153 30261 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54316: token expired
W20260926 22:52:01.063383 29729 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.148.62:40447)
I20260926 22:52:01.070158 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.063675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.13.148.62:40447)
0926 22:52:01.063709 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.063711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.063718 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:52:01.063762 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.063763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.063772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.063840 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:01.064379 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.065021 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.065988 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.066258 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.066315 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.067552 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:01.067553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:01.067553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:01.067560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:01.068926 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:01.069239 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:01.069240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:01.069240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:01.069292 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:01.069450 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:01.069451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:01.069451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:01.069587 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:01.069673 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:01.070006 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 22:52:01.070083 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:52:01.071262 29800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:52:01.072801 29800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:52:01.074946 29922 tablet_service.cc:1467] Processing CreateTablet for tablet f1715772e6bd4537ab74874040bd5954 (DEFAULT_TABLE table=test-table [id=def4474e1f5a40f39c8e0ce827ca3463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:01.075145 29922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1715772e6bd4537ab74874040bd5954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:01.076395 29880 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Bootstrap starting.
I20260926 22:52:01.076731 29880 tablet_bootstrap.cc:654] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:01.077237 29880 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: No bootstrap required, opened a new log
I20260926 22:52:01.077278 29880 ts_tablet_manager.cc:1397] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:01.077381 29880 raft_consensus.cc:348] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.077445 29880 raft_consensus.cc:374] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:01.077497 29880 raft_consensus.cc:729] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a027e875c596404399ef056dfea1ad23, State: Initialized, Role: FOLLOWER
I20260926 22:52:01.077538 29880 consensus_queue.cc:260] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.077615 29880 ts_tablet_manager.cc:1428] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:01.078991 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.074745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35517 (local address 127.0.0.1:47022)
0926 22:52:01.074971 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:01.074980 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:01.075026 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:01.075320 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:01.075321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:01.075323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:01.075370 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.075371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.076708 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.076709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.077973 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.077976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.078761 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.078762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.078827 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.078831 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:01.078877 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:52:01.078917 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":5,"thread_start_us":82,"threads_started":1}
I20260926 22:52:01.079085 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.074716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46743 (local address 127.0.0.1:43472)
0926 22:52:01.074762 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:01.074772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:01.074806 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:01.075329 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:01.075330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:01.075332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:01.075371 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.075372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.076763 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.076763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.077976 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.077978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.078862 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.078863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.078915 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.078918 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:01.078970 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:52:01.079003 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":63}
I20260926 22:52:01.079186 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.074803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.13.148.3:46743)
0926 22:52:01.075173 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.075176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.075185 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:52:01.075217 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.075218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.075232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.075477 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:01.075564 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.076684 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.078114 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.078761 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.078849 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.078909 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:01.078986 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 22:52:01.079018 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":106,"threads_started":1}
I20260926 22:52:01.079236 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.074957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.13.148.2:35517)
0926 22:52:01.075140 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.075144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.075156 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:52:01.075199 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.075201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.075216 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.075304 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:01.075429 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.076621 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.078067 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.078649 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.078716 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.078777 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:01.079116 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 22:52:01.079161 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260926 22:52:01.079798 30192 tablet_service.cc:1467] Processing CreateTablet for tablet f1715772e6bd4537ab74874040bd5954 (DEFAULT_TABLE table=test-table [id=def4474e1f5a40f39c8e0ce827ca3463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:01.079823 30060 tablet_service.cc:1467] Processing CreateTablet for tablet f1715772e6bd4537ab74874040bd5954 (DEFAULT_TABLE table=test-table [id=def4474e1f5a40f39c8e0ce827ca3463]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:01.080084 30192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1715772e6bd4537ab74874040bd5954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:01.080094 30060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1715772e6bd4537ab74874040bd5954. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:01.082883 30268 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Bootstrap starting.
I20260926 22:52:01.082891 30269 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Bootstrap starting.
I20260926 22:52:01.083420 30269 tablet_bootstrap.cc:654] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:01.083429 30268 tablet_bootstrap.cc:654] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:01.083891 30269 log.cc:824] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:01.083914 30268 log.cc:824] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:01.084581 30268 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: No bootstrap required, opened a new log
I20260926 22:52:01.084601 30269 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: No bootstrap required, opened a new log
I20260926 22:52:01.084654 30268 ts_tablet_manager.cc:1397] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:01.084656 30269 ts_tablet_manager.cc:1397] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:01.086001 30269 raft_consensus.cc:348] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.086024 30268 raft_consensus.cc:348] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.086144 30268 raft_consensus.cc:374] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:01.086138 30269 raft_consensus.cc:374] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:01.086165 30268 raft_consensus.cc:729] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dd4ab03f5af4c8184046fa0e3dc2c9d, State: Initialized, Role: FOLLOWER
I20260926 22:52:01.086169 30269 raft_consensus.cc:729] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1725d337d87947f8b625813f4956f648, State: Initialized, Role: FOLLOWER
I20260926 22:52:01.086278 30268 consensus_queue.cc:260] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.086282 30269 consensus_queue.cc:260] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.086534 30258 heartbeater.cc:503] Master 127.13.148.62:40447 was elected leader, sending a full tablet report...
I20260926 22:52:01.086534 30126 heartbeater.cc:503] Master 127.13.148.62:40447 was elected leader, sending a full tablet report...
I20260926 22:52:01.086544 30269 ts_tablet_manager.cc:1428] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:52:01.086768 30268 ts_tablet_manager.cc:1428] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260926 22:52:01.178579 30127 tablet.cc:2369] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:52:01.249665 29994 raft_consensus.cc:482] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:01.249840 29994 raft_consensus.cc:504] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.250236 29994 leader_election.cc:288] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743), 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
I20260926 22:52:01.253064 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.250528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:42893 (local address 127.13.148.3:46743)
0926 22:52:01.250573 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.250575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.250583 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:52:01.250609 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.250610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.250618 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.250666 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:01.251060 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.251539 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.252384 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.252751 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.252812 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.252820 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:01.252951 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 22:52:01.252995 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:52:01.253098 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.250406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46743 (local address 127.13.148.1:42893)
0926 22:52:01.250449 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:01.250461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:01.250527 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:01.250680 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:01.250680 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:01.250682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:01.250987 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.250990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.251610 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.251611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.252298 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.252301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.252819 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.252820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.252890 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.252892 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:52:01.252936 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:52:01.252983 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 22:52:01.253098 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.250543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:45653 (local address 127.13.148.2:35517)
0926 22:52:01.250591 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.250593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.250688 (+ 95us) server_negotiation.cc:407] Connection header received
0926 22:52:01.250739 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.250740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.250751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.250794 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:01.250926 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.251404 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.252382 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.252752 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.252813 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.252821 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:01.252959 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 22:52:01.252994 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:52:01.253186 30274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.250418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35517 (local address 127.13.148.1:45653)
0926 22:52:01.250622 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:01.250631 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:01.250689 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:01.250803 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:01.250803 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:01.250805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:01.250870 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.250872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.251476 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.251477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.252298 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:01.252300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.252839 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.252840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:01.252901 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.252904 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:01.252948 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:52:01.252987 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
I20260926 22:52:01.253412 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1715772e6bd4537ab74874040bd5954" candidate_uuid: "a027e875c596404399ef056dfea1ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" is_pre_election: true
I20260926 22:52:01.253487 30080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1715772e6bd4537ab74874040bd5954" candidate_uuid: "a027e875c596404399ef056dfea1ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1725d337d87947f8b625813f4956f648" is_pre_election: true
I20260926 22:52:01.253562 30212 raft_consensus.cc:2454] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a027e875c596404399ef056dfea1ad23 in term 0.
I20260926 22:52:01.253647 30080 raft_consensus.cc:2454] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a027e875c596404399ef056dfea1ad23 in term 0.
I20260926 22:52:01.253777 29874 leader_election.cc:302] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd4ab03f5af4c8184046fa0e3dc2c9d, a027e875c596404399ef056dfea1ad23; no voters:
I20260926 22:52:01.253911 29994 raft_consensus.cc:2781] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:52:01.253999 29994 raft_consensus.cc:482] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:52:01.254022 29994 raft_consensus.cc:3037] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:01.254705 29994 raft_consensus.cc:504] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.254825 29994 leader_election.cc:288] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 1 election: Requested vote from peers 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743), 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
I20260926 22:52:01.255013 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1715772e6bd4537ab74874040bd5954" candidate_uuid: "a027e875c596404399ef056dfea1ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d"
I20260926 22:52:01.255084 30212 raft_consensus.cc:3037] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:01.255776 30212 raft_consensus.cc:2454] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a027e875c596404399ef056dfea1ad23 in term 1.
I20260926 22:52:01.255878 30080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1715772e6bd4537ab74874040bd5954" candidate_uuid: "a027e875c596404399ef056dfea1ad23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1725d337d87947f8b625813f4956f648"
I20260926 22:52:01.255942 29874 leader_election.cc:302] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dd4ab03f5af4c8184046fa0e3dc2c9d, a027e875c596404399ef056dfea1ad23; no voters:
I20260926 22:52:01.255962 30080 raft_consensus.cc:3037] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:01.256152 29994 raft_consensus.cc:2781] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:01.256306 29994 raft_consensus.cc:686] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 LEADER]: Becoming Leader. State: Replica: a027e875c596404399ef056dfea1ad23, State: Running, Role: LEADER
I20260926 22:52:01.256379 29994 consensus_queue.cc:237] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:01.256829 30080 raft_consensus.cc:2454] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a027e875c596404399ef056dfea1ad23 in term 1.
I20260926 22:52:01.257150 29799 catalog_manager.cc:5495] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 reported cstate change: term changed from 0 to 1, leader changed from <none> to a027e875c596404399ef056dfea1ad23 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "a027e875c596404399ef056dfea1ad23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } health_report { overall_health: HEALTHY } } }
W20260926 22:52:01.298733 30259 tablet.cc:2369] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:52:01.302475 29989 tablet.cc:2369] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:52:01.304935 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.302828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53046 (local address 127.13.148.1:42169)
0926 22:52:01.302890 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:01.302893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:01.302903 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:52:01.302961 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:01.302961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:01.302978 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:01.303038 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:52:01.303155 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.303616 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.304331 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:01.304502 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:01.304542 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:01.304692 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:52:01.304787 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:52:01.304835 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:01.304864 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 22:52:01.304882 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 22:52:01.306354 30212 raft_consensus.cc:1261] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 1 FOLLOWER]: Refusing update from remote peer a027e875c596404399ef056dfea1ad23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:01.306355 30080 raft_consensus.cc:1261] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 1 FOLLOWER]: Refusing update from remote peer a027e875c596404399ef056dfea1ad23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:01.306643 30275 consensus_queue.cc:1035] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:01.306766 29994 consensus_queue.cc:1035] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:01.309083 30278 mvcc.cc:205] Tried to move back new op lower bound from 7333736944868474880 to 7333736944666640384. Current Snapshot: MvccSnapshot[applied={T|T < 7333736944868474880}]
I20260926 22:52:01.309096 30277 mvcc.cc:205] Tried to move back new op lower bound from 7333736944868474880 to 7333736944666640384. Current Snapshot: MvccSnapshot[applied={T|T < 7333736944868474880 or (T in {7333736944868474880})}]
I20260926 22:52:01.309257 30279 mvcc.cc:205] Tried to move back new op lower bound from 7333736944868474880 to 7333736944666640384. Current Snapshot: MvccSnapshot[applied={T|T < 7333736944868474880}]
I20260926 22:52:01.310940 29800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:01.311013 29800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:01.311661 29800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Sending DeleteTablet for 3 replicas of tablet f1715772e6bd4537ab74874040bd5954
I20260926 22:52:01.311955 30060 tablet_service.cc:1514] Processing DeleteTablet for tablet f1715772e6bd4537ab74874040bd5954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:47022
I20260926 22:52:01.311978 30192 tablet_service.cc:1514] Processing DeleteTablet for tablet f1715772e6bd4537ab74874040bd5954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43472
I20260926 22:52:01.312042 29922 tablet_service.cc:1514] Processing DeleteTablet for tablet f1715772e6bd4537ab74874040bd5954 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:01 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:53034
I20260926 22:52:01.312120 29997 tablet_replica.cc:330] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: stopping tablet replica
I20260926 22:52:01.312167 29997 raft_consensus.cc:2229] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:52:01.312223 30287 tablet_replica.cc:330] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: stopping tablet replica
I20260926 22:52:01.312244 29997 raft_consensus.cc:2258] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:01.312320 30287 raft_consensus.cc:2229] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:01.312418 30287 raft_consensus.cc:2258] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:01.312412 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 29838
I20260926 22:52:01.312561 30288 tablet_replica.cc:330] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: stopping tablet replica
I20260926 22:52:01.312636 30288 raft_consensus.cc:2229] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:01.312711 30288 raft_consensus.cc:2258] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:01.313266 30288 ts_tablet_manager.cc:1905] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:01.315042 30288 ts_tablet_manager.cc:1918] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:01.315104 30288 log.cc:1196] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/f1715772e6bd4537ab74874040bd5954
I20260926 22:52:01.315337 30288 ts_tablet_manager.cc:1939] T f1715772e6bd4537ab74874040bd5954 P 9dd4ab03f5af4c8184046fa0e3dc2c9d: Deleting consensus metadata
I20260926 22:52:01.315701 29784 catalog_manager.cc:4841] TS 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743): tablet f1715772e6bd4537ab74874040bd5954 (table test-table [id=def4474e1f5a40f39c8e0ce827ca3463]) successfully deleted
W20260926 22:52:01.317402 29786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:42169 (error 108)
I20260926 22:52:01.317708 30287 ts_tablet_manager.cc:1905] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:01.317865 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.317758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:42169 (local address 127.0.0.1:53048)
0926 22:52:01.317800 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:01.317835 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 22:52:01.318032 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 29992
W20260926 22:52:01.317997 29786 catalog_manager.cc:4584] TS a027e875c596404399ef056dfea1ad23 (127.13.148.1:42169): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1715772e6bd4537ab74874040bd5954: Network error: Client connection negotiation failed: client connection to 127.13.148.1:42169: connect: Connection refused (error 111)
I20260926 22:52:01.323318 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 30129
I20260926 22:52:01.323359 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:01.323255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35517 (local address 127.0.0.1:47032)
0926 22:52:01.323295 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:01.323329 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.2:35517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 22:52:01.327191 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 29767
I20260926 22:52:21.331681 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40447
--webserver_interface=127.13.148.62
--webserver_port=35301
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:40447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:21.409327 30290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:21.409518 30290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:21.409546 30290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:21.410701 30290 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 22:52:21.410739 30290 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 22:52:21.410755 30290 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 22:52:21.410770 30290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:21.410785 30290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:21.410812 30290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:21.410835 30290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:21.410851 30290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:21.412307 30290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:40447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:40447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=35301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:21.412542 30290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:21.412739 30290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:21.417258 30295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:21.417239 30298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:21.417371 30296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:21.417614 30290 server_base.cc:1034] running on GCE node
I20260926 22:52:21.417862 30290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:21.418131 30290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:21.419277 30290 hybrid_clock.cc:648] HybridClock initialized: now 1790463141419262 us; error 29 us; skew 500 ppm
Sep 26 22:52:21 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463141, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:21.425467 30290 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:52:21.425930 30290 webserver.cc:460] Webserver started at http://127.13.148.62:35301/ using document root <none> and password file <none>
I20260926 22:52:21.426102 30290 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:21.426141 30290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:21.427363 30290 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:21.427937 30305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:21.428078 30290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:21.428151 30290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "486bdb94e9d746a29fd1a2220c5f16b9"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "7edc63c12f0d27c037eb94769b89ce12"
server_key_iv: "65d884a2944666a2221349929da0bb63"
server_key_version: "encryptionkey@0"
I20260926 22:52:21.428483 30290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:21.444913 30290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:21.446636 30290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:21.446758 30290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:21.450446 30290 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:40447
I20260926 22:52:21.450481 30357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:40447 every 8 connection(s)
I20260926 22:52:21.450814 30290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:21.452172 30358 sys_catalog.cc:263] Verifying existing consensus state
I20260926 22:52:21.452744 30358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Bootstrap starting.
I20260926 22:52:21.456341 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 30290
I20260926 22:52:21.456569 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:42169
--local_ip_for_outbound_sockets=127.13.148.1
--tserver_master_addrs=127.13.148.62:40447
--webserver_port=41235
--webserver_interface=127.13.148.1
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:52:21.460711 30358 log.cc:824] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:21.462817 30358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:52:21.463069 30358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Bootstrap complete.
I20260926 22:52:21.465080 30358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:52:21.465304 30358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486bdb94e9d746a29fd1a2220c5f16b9, State: Initialized, Role: FOLLOWER
I20260926 22:52:21.465461 30358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:52:21.465535 30358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:21.465583 30358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:21.465628 30358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 2 FOLLOWER]: Advancing to term 3
I20260926 22:52:21.466590 30358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:52:21.466683 30358 leader_election.cc:302] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486bdb94e9d746a29fd1a2220c5f16b9; no voters:
I20260926 22:52:21.466835 30358 leader_election.cc:288] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 22:52:21.466894 30362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 22:52:21.467008 30362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [term 3 LEADER]: Becoming Leader. State: Replica: 486bdb94e9d746a29fd1a2220c5f16b9, State: Running, Role: LEADER
I20260926 22:52:21.467092 30358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:21.467098 30362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } }
I20260926 22:52:21.467406 30363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } } }
I20260926 22:52:21.467417 30364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486bdb94e9d746a29fd1a2220c5f16b9. Latest consensus state: current_term: 3 leader_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bdb94e9d746a29fd1a2220c5f16b9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 40447 } } }
I20260926 22:52:21.467496 30363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:21.467504 30364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:21.467689 30368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:21.468454 30368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:21.468562 30368 catalog_manager.cc:1259] Loaded cluster ID: ea5d5c8ef21945c4a62fb22dd74b8613
I20260926 22:52:21.468597 30368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:21.469873 30368 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:21.470036 30368 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Loaded TSK: 0
I20260926 22:52:21.470081 30368 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 486bdb94e9d746a29fd1a2220c5f16b9: Loaded TSK: 1
F20260926 22:52:21.470306 30368 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0294a7f9ad google::LogMessage::Fail() at ??:0
@ 0x7f0294a8288a google::LogMessage::SendToLog() at ??:0
@ 0x7f0294a7f4de google::LogMessage::Flush() at ??:0
@ 0x7f0294a80a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f02971ec104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f02971e9d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f029c9d91cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f029c9d948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f029c9d9594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f029c99fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f029c9a9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f029694d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0296942954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f02977a66db start_thread at ??:0
@ 0x7f029357d61f clone at ??:0
W20260926 22:52:21.550669 30360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:21.550868 30360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:21.550890 30360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:21.552071 30360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:21.552109 30360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:21.552135 30360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:52:21.553592 30360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:42169
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=41235
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:21.553829 30360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:21.554054 30360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:21.558774 30399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:21.558782 30402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:21.558807 30360 server_base.cc:1034] running on GCE node
W20260926 22:52:21.558962 30400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:21.559177 30360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:21.559459 30360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:21.560638 30360 hybrid_clock.cc:648] HybridClock initialized: now 1790463141560625 us; error 29 us; skew 500 ppm
Sep 26 22:52:21 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463141, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:21.563392 30360 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:52:21.563803 30360 webserver.cc:460] Webserver started at http://127.13.148.1:41235/ using document root <none> and password file <none>
I20260926 22:52:21.564011 30360 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:21.564057 30360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:21.565354 30360 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:52:21.565992 30409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:21.566131 30360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:21.566212 30360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "a027e875c596404399ef056dfea1ad23"
format_stamp: "Formatted at 2026-09-26 22:51:39 on dist-test-slave-98hb"
server_key: "09958293a812eb5dc25098c0246951cd"
server_key_iv: "ffb6d86c995b1c0fc87408ac27307380"
server_key_version: "encryptionkey@0"
I20260926 22:52:21.566516 30360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:21.573714 30360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:21.575601 30360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:21.575728 30360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:21.575909 30360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:21.576295 30416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 22:52:21.578118 30360 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 22:52:21.578157 30360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 22:52:21.578187 30360 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 22:52:21.578768 30360 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 22:52:21.578799 30360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 22:52:21.578881 30416 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Bootstrap starting.
I20260926 22:52:21.585049 30360 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:42169
I20260926 22:52:21.585178 30523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:42169 every 8 connection(s)
I20260926 22:52:21.585421 30360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:52:21.589486 30416 log.cc:824] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:21.591285 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 30360
I20260926 22:52:21.591536 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:35517
--local_ip_for_outbound_sockets=127.13.148.2
--tserver_master_addrs=127.13.148.62:40447
--webserver_port=39029
--webserver_interface=127.13.148.2
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:52:21.592029 30416 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 22:52:21.592319 30416 tablet_bootstrap.cc:492] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Bootstrap complete.
I20260926 22:52:21.593376 30416 ts_tablet_manager.cc:1397] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260926 22:52:21.594785 30416 raft_consensus.cc:348] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:21.595099 30416 raft_consensus.cc:729] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a027e875c596404399ef056dfea1ad23, State: Initialized, Role: FOLLOWER
I20260926 22:52:21.595271 30416 consensus_queue.cc:260] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:21.595548 30416 ts_tablet_manager.cc:1428] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260926 22:52:21.655320 30526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:52:21.585995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:52607)
0926 22:52:21.586148 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.586187 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:21.586618 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:21.655105 (+ 68487us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
W20260926 22:52:21.655576 30524 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:40447 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:40447: Client connection negotiation failed: client connection to 127.13.148.62:40447: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 22:52:21.655920 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:21.655805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:54035)
0926 22:52:21.655855 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.655895 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 22:52:21.656239 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:21.656146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:48375)
0926 22:52:21.656190 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.656218 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 22:52:21.656342 30524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:52:21.669880 30528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:21.670060 30528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:21.670081 30528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:21.671171 30528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:21.671203 30528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:21.671226 30528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:52:21.672585 30528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:35517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=39029
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:21.672766 30528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:21.672940 30528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:21.677515 30534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:21.677507 30535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:21.677507 30537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:21.677697 30528 server_base.cc:1034] running on GCE node
I20260926 22:52:21.677901 30528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:21.678099 30528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:21.679235 30528 hybrid_clock.cc:648] HybridClock initialized: now 1790463141679224 us; error 28 us; skew 500 ppm
Sep 26 22:52:21 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463141, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:21.681463 30528 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:52:21.681898 30528 webserver.cc:460] Webserver started at http://127.13.148.2:39029/ using document root <none> and password file <none>
I20260926 22:52:21.682111 30528 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:21.682168 30528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:21.683367 30528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:21.683983 30544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:21.684144 30528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:21.684239 30528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "1725d337d87947f8b625813f4956f648"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "1fb8827e95ab73d056c0526299314cb8"
server_key_iv: "1a1a55a15a95854e39688b426771efcd"
server_key_version: "encryptionkey@0"
I20260926 22:52:21.684551 30528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:21.692469 30528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:21.694259 30528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:21.694379 30528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:21.694554 30528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:21.694900 30551 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 22:52:21.696220 30551 ts_tablet_manager.cc:1852] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 22:52:21.696483 30551 ts_tablet_manager.cc:1905] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:21.698560 30551 ts_tablet_manager.cc:1918] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:21.698633 30551 log.cc:1196] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/f1715772e6bd4537ab74874040bd5954
I20260926 22:52:21.698868 30551 ts_tablet_manager.cc:1939] T f1715772e6bd4537ab74874040bd5954 P 1725d337d87947f8b625813f4956f648: Deleting consensus metadata
I20260926 22:52:21.699045 30528 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 22:52:21.699091 30528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 22:52:21.699128 30528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:21.699151 30528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:21.704588 30528 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:35517
I20260926 22:52:21.704654 30658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:35517 every 8 connection(s)
I20260926 22:52:21.704901 30528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:52:21.705832 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:21.705451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:40165)
0926 22:52:21.705589 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.705653 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
W20260926 22:52:21.706022 30659 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:40447 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:40447: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
I20260926 22:52:21.706022 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 30528
I20260926 22:52:21.706256 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:46743
--local_ip_for_outbound_sockets=127.13.148.3
--tserver_master_addrs=127.13.148.62:40447
--webserver_port=43341
--webserver_interface=127.13.148.3
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:37245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:52:21.706297 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:21.706213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:46139)
0926 22:52:21.706254 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.706278 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 22:52:21.706565 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:21.706452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:47251)
0926 22:52:21.706511 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.706542 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 22:52:21.706683 30659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 22:52:21.784303 30662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:21.784502 30662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:21.784523 30662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:21.785646 30662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:21.785686 30662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:21.785710 30662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:52:21.787153 30662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:46743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=43341
--tserver_master_addrs=127.13.148.62:40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:21.787330 30662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:21.787497 30662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:21.792021 30668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:21.792057 30667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:21.792060 30662 server_base.cc:1034] running on GCE node
W20260926 22:52:21.792028 30670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:21.792364 30662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:21.792570 30662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:21.793701 30662 hybrid_clock.cc:648] HybridClock initialized: now 1790463141793689 us; error 26 us; skew 500 ppm
Sep 26 22:52:21 dist-test-slave-98hb krb5kdc[29220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463141, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:21.795862 30662 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:52:21.796267 30662 webserver.cc:460] Webserver started at http://127.13.148.3:43341/ using document root <none> and password file <none>
I20260926 22:52:21.796463 30662 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:21.796509 30662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:21.797688 30662 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:52:21.798206 30677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:21.798342 30662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:21.798411 30662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d"
format_stamp: "Formatted at 2026-09-26 22:51:40 on dist-test-slave-98hb"
server_key: "ab1dd394854b203e718053824ff8b004"
server_key_iv: "28c7a856dd03a515eda08eaaad51657b"
server_key_version: "encryptionkey@0"
I20260926 22:52:21.798705 30662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:21.806401 30662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:21.807927 30662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:21.808043 30662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:21.808238 30662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:21.808486 30662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:21.808517 30662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:21.808537 30662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:21.808559 30662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:21.813624 30662 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:46743
I20260926 22:52:21.813699 30790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:46743 every 8 connection(s)
I20260926 22:52:21.813982 30662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:52:21.814906 30793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:21.814571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:46545)
0926 22:52:21.814702 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.814764 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
W20260926 22:52:21.815274 30791 heartbeater.cc:644] Failed to heartbeat to 127.13.148.62:40447 (0 consecutive failures): Network error: Failed to ping master at 127.13.148.62:40447: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
I20260926 22:52:21.815511 30793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:21.815431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:33991)
0926 22:52:21.815469 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.815493 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 22:52:21.815737 30793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:21.815671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:43525)
0926 22:52:21.815703 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:21.815723 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 22:52:21.815819 30791 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 22:52:21.820576 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 30662
I20260926 22:52:22.656872 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:22.656652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:44281)
0926 22:52:22.656775 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:22.656812 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 22:52:22.707118 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:22.706956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:39707)
0926 22:52:22.707057 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:22.707086 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260926 22:52:22.739985 30797 raft_consensus.cc:482] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:22.740100 30797 raft_consensus.cc:504] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:22.740366 30797 leader_election.cc:288] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743), 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
Sep 26 22:52:22 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463141, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.3@KRBTEST.COM
Sep 26 22:52:22 dist-test-slave-98hb krb5kdc[29220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463141, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.2@KRBTEST.COM
I20260926 22:52:22.749775 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:22.740510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:46743 (local address 127.13.148.1:36823)
0926 22:52:22.740566 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:22.740577 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:22.740631 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:22.741443 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:22.741447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:22.741909 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:22.742106 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:22.742111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:22.743233 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:22.743234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:22.743993 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:22.743995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:22.744524 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:22.744525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:22.744637 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:22.745121 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:22.745129 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:22.746519 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:22.748444 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:22.748450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:22.748452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:22.748646 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:22.748860 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:22.748860 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:22.748861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:22.748951 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:22.749287 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:22.749292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:22.749412 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 22:52:22.749496 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 22:52:22.749642 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":48}
I20260926 22:52:22.749785 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:22.740641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:36823 (local address 127.13.148.3:46743)
0926 22:52:22.740794 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:22.740803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:22.740820 (+ 17us) server_negotiation.cc:407] Connection header received
0926 22:52:22.741214 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:22.741216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:22.741267 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:22.741374 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:22.742212 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:22.743120 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:22.744067 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:22.744448 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:22.744539 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:22.746613 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:22.746617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:22.746617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:22.746628 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:22.748380 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:22.748715 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:22.748718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:22.748718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:22.748806 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:22.749004 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:22.749004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:22.749005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:22.749227 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:22.749295 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:22.749552 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 22:52:22.749674 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":52,"threads_started":1}
I20260926 22:52:22.750591 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:22.740526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:35517 (local address 127.13.148.1:47129)
0926 22:52:22.740646 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:22.740655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:22.740704 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:22.741482 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:22.741483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:22.741909 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:22.742106 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:22.742111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:22.743199 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:22.743200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:22.743992 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:22.743994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:22.744521 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:22.744522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:22.744640 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:22.745166 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:22.745171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:22.746827 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:22.749315 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:22.749316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:22.749316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:22.749475 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:22.749681 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:22.749682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:22.749682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:22.749762 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:22.750331 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:22.750332 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:22.750365 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 22:52:22.750423 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:52:22.750512 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":30,"thread_start_us":55,"threads_started":1}
W20260926 22:52:22.750594 30410 leader_election.cc:341] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743): Not found: Tablet not found: f1715772e6bd4537ab74874040bd5954
I20260926 22:52:22.750895 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:22.740616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:47129 (local address 127.13.148.2:35517)
0926 22:52:22.740759 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:22.740772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:22.740796 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:52:22.741309 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:22.741312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:22.741355 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:22.741486 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:22.742211 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:22.743108 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:22.744073 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:22.744441 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:22.744539 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:22.746914 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:22.746918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:22.746918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:22.746929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:22.749240 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:22.749544 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:22.749546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:22.749547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:22.749624 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:22.749807 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:22.749807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:22.749808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:22.750260 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:22.750340 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:22.750486 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 22:52:22.750623 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
W20260926 22:52:22.751408 30413 leader_election.cc:341] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517): Not found: Tablet not found: f1715772e6bd4537ab74874040bd5954
I20260926 22:52:22.751463 30413 leader_election.cc:302] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a027e875c596404399ef056dfea1ad23; no voters: 1725d337d87947f8b625813f4956f648, 9dd4ab03f5af4c8184046fa0e3dc2c9d
I20260926 22:52:22.751560 30797 raft_consensus.cc:2726] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:52:22.816272 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:22.816086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:53893)
0926 22:52:22.816201 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:22.816236 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260926 22:52:23.657431 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:23.657238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:40229)
0926 22:52:23.657349 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:23.657385 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260926 22:52:23.707655 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:23.707470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:49311)
0926 22:52:23.707584 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:23.707617 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260926 22:52:23.816848 30804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:23.816656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:59085)
0926 22:52:23.816776 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:23.816811 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260926 22:52:24.421761 30805 raft_consensus.cc:482] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:24.421851 30805 raft_consensus.cc:504] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:24.422009 30805 leader_election.cc:288] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743), 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
W20260926 22:52:24.422370 30413 leader_election.cc:341] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517): Not found: Tablet not found: f1715772e6bd4537ab74874040bd5954
W20260926 22:52:24.422451 30410 leader_election.cc:341] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743): Not found: Tablet not found: f1715772e6bd4537ab74874040bd5954
I20260926 22:52:24.422526 30410 leader_election.cc:302] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a027e875c596404399ef056dfea1ad23; no voters: 1725d337d87947f8b625813f4956f648, 9dd4ab03f5af4c8184046fa0e3dc2c9d
I20260926 22:52:24.422592 30805 raft_consensus.cc:2726] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:52:24.658006 30806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:24.657829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:55079)
0926 22:52:24.657930 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:24.657960 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260926 22:52:24.708197 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:24.708019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:47825)
0926 22:52:24.708122 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:24.708151 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260926 22:52:24.817341 30808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:24.817175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:38109)
0926 22:52:24.817278 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:24.817304 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260926 22:52:25.658584 30809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:25.658368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:59349)
0926 22:52:25.658499 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:25.658535 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260926 22:52:25.708755 30810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:25.708566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:51583)
0926 22:52:25.708679 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:25.708715 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260926 22:52:25.817874 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:25.817672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:56603)
0926 22:52:25.817807 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:25.817838 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260926 22:52:26.659209 30812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:26.658993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:54939)
0926 22:52:26.659123 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:26.659159 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260926 22:52:26.709267 30813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:26.709073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:58859)
0926 22:52:26.709199 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:26.709230 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260926 22:52:26.818436 30814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:26.818232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:48071)
0926 22:52:26.818356 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:26.818396 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":53,"threads_started":1}
I20260926 22:52:27.659812 30815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:27.659614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:34755)
0926 22:52:27.659733 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:27.659768 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260926 22:52:27.689266 30816 raft_consensus.cc:482] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:27.689354 30816 raft_consensus.cc:504] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:27.689488 30816 leader_election.cc:288] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743), 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
W20260926 22:52:27.689837 30410 leader_election.cc:341] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743): Not found: Tablet not found: f1715772e6bd4537ab74874040bd5954
W20260926 22:52:27.689917 30413 leader_election.cc:341] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517): Not found: Tablet not found: f1715772e6bd4537ab74874040bd5954
I20260926 22:52:27.689975 30413 leader_election.cc:302] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a027e875c596404399ef056dfea1ad23; no voters: 1725d337d87947f8b625813f4956f648, 9dd4ab03f5af4c8184046fa0e3dc2c9d
I20260926 22:52:27.690049 30816 raft_consensus.cc:2726] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:52:27.709803 30817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:27.709625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:47397)
0926 22:52:27.709724 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:27.709768 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260926 22:52:27.819013 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:27.818813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:57947)
0926 22:52:27.818935 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:27.818973 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260926 22:52:28.660449 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:28.660212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:52467)
0926 22:52:28.660364 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:28.660401 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260926 22:52:28.710376 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:28.710186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:49981)
0926 22:52:28.710301 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:28.710334 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260926 22:52:28.819533 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:28.819343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:52853)
0926 22:52:28.819463 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:28.819494 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260926 22:52:29.661104 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:29.660911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:43343)
0926 22:52:29.661029 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:29.661060 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260926 22:52:29.710934 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:29.710755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:34839)
0926 22:52:29.710861 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:29.710893 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260926 22:52:29.820135 30824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:29.819951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:56485)
0926 22:52:29.820065 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:29.820098 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260926 22:52:30.661689 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:30.661488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:35665)
0926 22:52:30.661609 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:30.661639 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260926 22:52:30.711516 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:30.711325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:51255)
0926 22:52:30.711446 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:30.711477 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260926 22:52:30.820683 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:30.820490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:56947)
0926 22:52:30.820609 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:30.820643 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260926 22:52:31.662302 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:31.662076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:35461)
0926 22:52:31.662221 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:31.662256 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260926 22:52:31.712114 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:31.711905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:33199)
0926 22:52:31.712042 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:31.712075 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260926 22:52:31.821228 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:31.821037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:42557)
0926 22:52:31.821154 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:31.821186 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260926 22:52:32.201191 30831 raft_consensus.cc:482] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:32.201278 30831 raft_consensus.cc:504] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dd4ab03f5af4c8184046fa0e3dc2c9d" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 46743 } } peers { permanent_uuid: "1725d337d87947f8b625813f4956f648" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 35517 } } peers { permanent_uuid: "a027e875c596404399ef056dfea1ad23" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 42169 } }
I20260926 22:52:32.201436 30831 leader_election.cc:288] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743), 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517)
W20260926 22:52:32.201781 30410 leader_election.cc:341] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9dd4ab03f5af4c8184046fa0e3dc2c9d (127.13.148.3:46743): Not found: Tablet not found: f1715772e6bd4537ab74874040bd5954
W20260926 22:52:32.201867 30413 leader_election.cc:341] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1725d337d87947f8b625813f4956f648 (127.13.148.2:35517): Not found: Tablet not found: f1715772e6bd4537ab74874040bd5954
I20260926 22:52:32.201907 30413 leader_election.cc:302] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a027e875c596404399ef056dfea1ad23; no voters: 1725d337d87947f8b625813f4956f648, 9dd4ab03f5af4c8184046fa0e3dc2c9d
I20260926 22:52:32.201983 30831 raft_consensus.cc:2726] T f1715772e6bd4537ab74874040bd5954 P a027e875c596404399ef056dfea1ad23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 22:52:32.662896 30832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:32.662684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:57663)
0926 22:52:32.662808 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:32.662842 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260926 22:52:32.712692 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:32.712513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:41467)
0926 22:52:32.712620 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:32.712652 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260926 22:52:32.821843 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:32.821633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:46697)
0926 22:52:32.821770 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:32.821803 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260926 22:52:33.663435 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:33.663242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:49871)
0926 22:52:33.663355 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:33.663390 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260926 22:52:33.713238 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:33.713055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:36879)
0926 22:52:33.713163 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:33.713197 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260926 22:52:33.822376 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:33.822198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:53875)
0926 22:52:33.822300 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:33.822334 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
I20260926 22:52:34.663993 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:34.663804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:57679)
0926 22:52:34.663911 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:34.663946 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260926 22:52:34.713845 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:34.713638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:40059)
0926 22:52:34.713772 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:34.713803 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260926 22:52:34.822934 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:34.822749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:48711)
0926 22:52:34.822862 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:34.822892 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260926 22:52:35.664588 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:35.664372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:33437)
0926 22:52:35.664504 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:35.664541 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260926 22:52:35.714480 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:35.714236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:58723)
0926 22:52:35.714395 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:35.714436 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260926 22:52:35.823508 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:35.823315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:58393)
0926 22:52:35.823434 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:35.823467 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260926 22:52:36.665292 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:36.665078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.1:57921)
0926 22:52:36.665208 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:36.665248 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260926 22:52:36.715191 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:36.714997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.2:44767)
0926 22:52:36.715116 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:36.715150 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260926 22:52:36.822705 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 30360
I20260926 22:52:36.824146 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:36.823930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:40447 (local address 127.13.148.3:37101)
0926 22:52:36.824037 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:36.824068 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.62:40447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260926 22:52:36.826926 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 30528
I20260926 22:52:36.830598 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 30662
2026-09-26T22:52:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58151 ms)
[----------] 30 tests from SecurityITest (221423 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:36 dist-test-slave-98hb krb5kdc[30849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:36 dist-test-slave-98hb krb5kdc[30849](info): set up 2 sockets
Sep 26 22:52:36 dist-test-slave-98hb krb5kdc[30849](info): commencing operation
krb5kdc: starting...
W20260926 22:52:37.777554 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:37 dist-test-slave-98hb krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:52:37.884120 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39995
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:37.960112 30862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:37.960309 30862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:37.960330 30862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:37.961441 30862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:37.961479 30862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:37.961493 30862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:37.961503 30862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:37.961514 30862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:37.962970 30862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:37.963186 30862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:37.963387 30862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:37.967994 30867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:37.968034 30862 server_base.cc:1034] running on GCE node
W20260926 22:52:37.968094 30868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:37.968034 30870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:37.968636 30862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:37.968887 30862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:37.970011 30862 hybrid_clock.cc:648] HybridClock initialized: now 1790463157969998 us; error 32 us; skew 500 ppm
Sep 26 22:52:37 dist-test-slave-98hb krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463157, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:37.972251 30862 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:52:37.972644 30862 webserver.cc:460] Webserver started at http://127.13.148.62:41391/ using document root <none> and password file <none>
I20260926 22:52:37.972827 30862 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:37.972873 30862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:37.972992 30862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:37.974020 30862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "15058ea3b04a4fe6b17aa8d40e321bcc"
format_stamp: "Formatted at 2026-09-26 22:52:37 on dist-test-slave-98hb"
server_key: "b96752c6dcb2612a5f9e5951c63c4e81"
server_key_iv: "72212db4eab793d982ccba3ba5e607bc"
server_key_version: "encryptionkey@0"
I20260926 22:52:37.974332 30862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "15058ea3b04a4fe6b17aa8d40e321bcc"
format_stamp: "Formatted at 2026-09-26 22:52:37 on dist-test-slave-98hb"
server_key: "b96752c6dcb2612a5f9e5951c63c4e81"
server_key_iv: "72212db4eab793d982ccba3ba5e607bc"
server_key_version: "encryptionkey@0"
I20260926 22:52:37.975502 30862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:37.976141 30877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:37.976284 30862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:37.976359 30862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "15058ea3b04a4fe6b17aa8d40e321bcc"
format_stamp: "Formatted at 2026-09-26 22:52:37 on dist-test-slave-98hb"
server_key: "b96752c6dcb2612a5f9e5951c63c4e81"
server_key_iv: "72212db4eab793d982ccba3ba5e607bc"
server_key_version: "encryptionkey@0"
I20260926 22:52:37.976414 30862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:37.986431 30862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:37.988293 30862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:37.988418 30862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:37.991946 30862 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39995
I20260926 22:52:37.991976 30929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39995 every 8 connection(s)
I20260926 22:52:37.992275 30862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:37.993014 30930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:37.995208 30930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc: Bootstrap starting.
I20260926 22:52:37.995718 30930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:37.995934 30930 log.cc:824] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:37.996529 30930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc: No bootstrap required, opened a new log
I20260926 22:52:37.997906 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 30862
I20260926 22:52:37.997993 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:37.998174 13904 external_mini_cluster.cc:1428] Setting key 934d78ecf6984b0075b4737bec1664ab
I20260926 22:52:37.998101 30930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15058ea3b04a4fe6b17aa8d40e321bcc" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39995 } }
I20260926 22:52:37.998222 30930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:37.998262 30930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15058ea3b04a4fe6b17aa8d40e321bcc, State: Initialized, Role: FOLLOWER
I20260926 22:52:37.998396 30930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15058ea3b04a4fe6b17aa8d40e321bcc" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39995 } }
I20260926 22:52:37.998476 30930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:37.998512 30930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:37.998574 30930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:37.999482 30930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15058ea3b04a4fe6b17aa8d40e321bcc" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39995 } }
I20260926 22:52:37.999608 30930 leader_election.cc:302] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15058ea3b04a4fe6b17aa8d40e321bcc; no voters:
I20260926 22:52:37.999764 30930 leader_election.cc:288] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:37.999788 30935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:37.999935 30935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [term 1 LEADER]: Becoming Leader. State: Replica: 15058ea3b04a4fe6b17aa8d40e321bcc, State: Running, Role: LEADER
I20260926 22:52:37.999998 30930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:38.000052 30935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15058ea3b04a4fe6b17aa8d40e321bcc" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39995 } }
I20260926 22:52:38.000418 30936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15058ea3b04a4fe6b17aa8d40e321bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15058ea3b04a4fe6b17aa8d40e321bcc" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39995 } } }
I20260926 22:52:38.000497 30936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:38.000646 30937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15058ea3b04a4fe6b17aa8d40e321bcc. Latest consensus state: current_term: 1 leader_uuid: "15058ea3b04a4fe6b17aa8d40e321bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15058ea3b04a4fe6b17aa8d40e321bcc" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39995 } } }
I20260926 22:52:38.000707 30937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:38.001080 30943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:38.001570 30943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:38.002964 30943 catalog_manager.cc:1347] Generated new cluster ID: 29ec875303ef4636becd3f4087633915
I20260926 22:52:38.003018 30943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:52:38 dist-test-slave-98hb krb5kdc[30849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:38.008059 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:37.998585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.13.148.62:39995)
0926 22:52:37.998757 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:37.998766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:37.998861 (+ 95us) server_negotiation.cc:407] Connection header received
0926 22:52:37.998999 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:37.999004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:37.999044 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:37.999151 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:37.999897 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.001381 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.002490 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.002982 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.003361 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.005218 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:38.005222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:38.005222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:38.005231 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:38.006755 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.007055 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.007058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.007058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.007148 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.007322 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.007323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.007323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.007542 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:38.007686 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.007783 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.007912 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260926 22:52:38.026893 30943 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:38.027593 30943 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:38.041270 30943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc: Generated new TSK 0
I20260926 22:52:38.041445 30943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:52:38.068514 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39995
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:38.145627 30956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:38.145855 30956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:38.145881 30956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:38.146986 30956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:38.147022 30956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:38.147049 30956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:52:38.148425 30956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:38.148628 30956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:38.148844 30956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:38.153352 30962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.153369 30956 server_base.cc:1034] running on GCE node
W20260926 22:52:38.153468 30964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:38.153378 30961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.153725 30956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:38.153981 30956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:38.155118 30956 hybrid_clock.cc:648] HybridClock initialized: now 1790463158155112 us; error 32 us; skew 500 ppm
Sep 26 22:52:38 dist-test-slave-98hb krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:38.157467 30956 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:52:38.157938 30956 webserver.cc:460] Webserver started at http://127.13.148.1:33065/ using document root <none> and password file <none>
I20260926 22:52:38.158124 30956 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:38.158169 30956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:38.158288 30956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:38.159238 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "8d998f03bc46441f9dc3d6456f1bf544"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "0fb42e1267d7dfb9c0c427f321667a10"
server_key_iv: "818f27e3080dac0c09360e34e1e694bb"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.159547 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "8d998f03bc46441f9dc3d6456f1bf544"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "0fb42e1267d7dfb9c0c427f321667a10"
server_key_iv: "818f27e3080dac0c09360e34e1e694bb"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.160712 30956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:52:38.161294 30971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.161437 30956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:38.161512 30956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "8d998f03bc46441f9dc3d6456f1bf544"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "0fb42e1267d7dfb9c0c427f321667a10"
server_key_iv: "818f27e3080dac0c09360e34e1e694bb"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.161576 30956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:38.176965 30956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:38.178731 30956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:38.178861 30956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:38.179049 30956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:38.179327 30956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:38.179360 30956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.179388 30956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:38.179412 30956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.184682 30956 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:36957
I20260926 22:52:38.184728 31084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:36957 every 8 connection(s)
I20260926 22:52:38.185006 30956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:52:38 dist-test-slave-98hb krb5kdc[30849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:38.192687 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 30956
I20260926 22:52:38.192778 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:52:38.192986 13904 external_mini_cluster.cc:1428] Setting key 259e04384dfdf593eaee0dd90b4c503a
I20260926 22:52:38.193287 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.185630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:33663 (local address 127.13.148.62:39995)
0926 22:52:38.185687 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.185689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.185915 (+ 226us) server_negotiation.cc:407] Connection header received
0926 22:52:38.186394 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.186395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.186409 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.186458 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:38.187307 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.187803 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.188814 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.189007 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.189054 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.190979 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:38.190980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:38.190981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:38.190987 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:38.192001 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.192341 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.192342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.192342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.192393 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.192616 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.192617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.192617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.192733 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:38.192807 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.193110 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.193197 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:52:38.193431 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.185616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39995 (local address 127.13.148.1:33663)
0926 22:52:38.185797 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.185837 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.186316 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.186516 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.186519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.187048 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:38.187242 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.187246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.187862 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.187863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.188748 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.188750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.189080 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.189081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.189173 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.189601 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:38.189609 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:38.190898 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:38.192072 (+ 1174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:38.192078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:38.192080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:38.192278 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:38.192443 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:38.192444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:38.192444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:38.192548 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:38.192803 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:38.192809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:38.192963 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.193070 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.193216 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260926 22:52:38.193850 31085 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39995
I20260926 22:52:38.193943 31085 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:38.194103 31085 heartbeater.cc:511] Master 127.13.148.62:39995 requested a full tablet report, sending...
I20260926 22:52:38.194557 30894 ts_manager.cc:194] Registered new tserver with Master: 8d998f03bc46441f9dc3d6456f1bf544 (127.13.148.1:36957)
I20260926 22:52:38.195325 30894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:33663
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:52:38.214174 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39995
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:38.290580 31090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:38.290777 31090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:38.290798 31090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:38.291903 31090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:38.291940 31090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:38.291967 31090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:52:38.293396 31090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:38.293601 31090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:38.293833 31090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:38.298314 31096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.298301 31090 server_base.cc:1034] running on GCE node
W20260926 22:52:38.298280 31095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:38.298278 31098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.298707 31090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:38.298933 31090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:38.300077 31090 hybrid_clock.cc:648] HybridClock initialized: now 1790463158300063 us; error 28 us; skew 500 ppm
Sep 26 22:52:38 dist-test-slave-98hb krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:38.302341 31090 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:52:38.302693 31090 webserver.cc:460] Webserver started at http://127.13.148.2:43347/ using document root <none> and password file <none>
I20260926 22:52:38.302870 31090 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:38.302907 31090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:38.302992 31090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:38.303931 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "cf44920de0c246ee95c290199b1cb8b5"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "9c1da2fb7c81b418ca4203c6717004cc"
server_key_iv: "92dbdf7213dd3978bfe95cce5b7bc6a5"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.304219 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "cf44920de0c246ee95c290199b1cb8b5"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "9c1da2fb7c81b418ca4203c6717004cc"
server_key_iv: "92dbdf7213dd3978bfe95cce5b7bc6a5"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.305316 31090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:38.305961 31105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.306121 31090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:38.306205 31090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "cf44920de0c246ee95c290199b1cb8b5"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "9c1da2fb7c81b418ca4203c6717004cc"
server_key_iv: "92dbdf7213dd3978bfe95cce5b7bc6a5"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.306270 31090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:38.318212 31090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:38.319783 31090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:38.319895 31090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:38.320093 31090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:38.320364 31090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:38.320395 31090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.320416 31090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:38.320439 31090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.325477 31090 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:45863
I20260926 22:52:38.325543 31218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:45863 every 8 connection(s)
I20260926 22:52:38.325840 31090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:52:38.327911 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 31090
I20260926 22:52:38.328009 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:52:38.328212 13904 external_mini_cluster.cc:1428] Setting key b63788d156ab9e32e06829ec5b5a2ee6
Sep 26 22:52:38 dist-test-slave-98hb krb5kdc[30849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:38.334832 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.326521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:42507 (local address 127.13.148.62:39995)
0926 22:52:38.326567 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.326569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.326757 (+ 188us) server_negotiation.cc:407] Connection header received
0926 22:52:38.327189 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.327191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.327200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.327247 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:38.328116 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.328772 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.330129 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.330328 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.330384 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.332431 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:38.332432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:38.332433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:38.332440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:38.333636 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.334013 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.334014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.334014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.334064 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.334271 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.334271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.334272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.334373 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:38.334426 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.334670 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.334745 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:52:38.334993 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.326484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39995 (local address 127.13.148.2:42507)
0926 22:52:38.326645 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.326680 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.327116 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.327299 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.327301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.327818 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:38.327996 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.328001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.328868 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.328869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.330047 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.330051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.330401 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.330402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.330513 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.330941 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:38.330948 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:38.332349 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:38.333717 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:38.333723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:38.333725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:38.333950 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:38.334115 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:38.334116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:38.334116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:38.334218 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:38.334424 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:38.334430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:38.334552 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.334637 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.334787 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260926 22:52:38.335481 31219 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39995
I20260926 22:52:38.335557 31219 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:38.335733 31219 heartbeater.cc:511] Master 127.13.148.62:39995 requested a full tablet report, sending...
I20260926 22:52:38.336092 30894 ts_manager.cc:194] Registered new tserver with Master: cf44920de0c246ee95c290199b1cb8b5 (127.13.148.2:45863)
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
I20260926 22:52:38.336558 30894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:42507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:52:38.352682 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39995
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:38.429811 31224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:38.429997 31224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:38.430019 31224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:38.431087 31224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:38.431118 31224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:38.431141 31224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:52:38.432487 31224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:38.432672 31224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:38.432838 31224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:38.437500 31229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.437520 31224 server_base.cc:1034] running on GCE node
W20260926 22:52:38.437484 31232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:38.437475 31230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.437882 31224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:38.438113 31224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:38.439241 31224 hybrid_clock.cc:648] HybridClock initialized: now 1790463158439231 us; error 28 us; skew 500 ppm
Sep 26 22:52:38 dist-test-slave-98hb krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:38.441557 31224 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:52:38.441985 31224 webserver.cc:460] Webserver started at http://127.13.148.3:41025/ using document root <none> and password file <none>
I20260926 22:52:38.442186 31224 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:38.442232 31224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:38.442350 31224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:38.443321 31224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "2cb7d3ca3655411c822d3095e1352534"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "4b2e077f7c75ce50007fe742c9942ec2"
server_key_iv: "d86a971ffe30f4e057747c41598601b6"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.443634 31224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "2cb7d3ca3655411c822d3095e1352534"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "4b2e077f7c75ce50007fe742c9942ec2"
server_key_iv: "d86a971ffe30f4e057747c41598601b6"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.444834 31224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:38.445472 31239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.445603 31224 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:38.445680 31224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "2cb7d3ca3655411c822d3095e1352534"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "4b2e077f7c75ce50007fe742c9942ec2"
server_key_iv: "d86a971ffe30f4e057747c41598601b6"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.445761 31224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:38.473856 31224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:38.475925 31224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:38.476084 31224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:38.476294 31224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:38.476595 31224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:38.476634 31224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.476661 31224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:38.476678 31224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.482070 31224 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:45705
I20260926 22:52:38.482118 31352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:45705 every 8 connection(s)
I20260926 22:52:38.482401 31224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:52:38.486908 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 31224
I20260926 22:52:38.487026 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:52:38.487208 13904 external_mini_cluster.cc:1428] Setting key 61042d55565fe47a2a55cd68e3be04e8
Sep 26 22:52:38 dist-test-slave-98hb krb5kdc[30849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463158, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:38.490530 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.482990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:45617 (local address 127.13.148.62:39995)
0926 22:52:38.483060 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.483062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.483266 (+ 204us) server_negotiation.cc:407] Connection header received
0926 22:52:38.483724 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.483725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.483736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.483773 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:38.484622 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.485102 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.486113 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.486301 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.486359 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.488339 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:38.488340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:38.488340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:38.488346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:38.489413 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.489797 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.489799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.489799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.489858 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.490051 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.490052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.490052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.490149 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:38.490197 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.490393 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.490454 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 22:52:38.490675 31355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.482971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39995 (local address 127.13.148.3:45617)
0926 22:52:38.483157 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.483193 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.483634 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.483836 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.483839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.484365 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:38.484553 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.484558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.485164 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.485165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.485994 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.485996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.486372 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.486372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.486464 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.486873 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:38.486881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:38.488268 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:38.489482 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:38.489487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:38.489489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:38.489697 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:38.489916 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:38.489917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:38.489917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:38.490007 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:38.490192 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:38.490195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:38.490296 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.490369 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.490478 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260926 22:52:38.491106 31353 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39995
I20260926 22:52:38.491189 31353 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:38.491319 31353 heartbeater.cc:511] Master 127.13.148.62:39995 requested a full tablet report, sending...
I20260926 22:52:38.491616 30894 ts_manager.cc:194] Registered new tserver with Master: 2cb7d3ca3655411c822d3095e1352534 (127.13.148.3:45705)
I20260926 22:52:38.491933 30894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:45617
I20260926 22:52:38.498735 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:52:38.505270 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.500071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.13.148.62:39995)
0926 22:52:38.500127 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.500129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.500198 (+ 69us) server_negotiation.cc:407] Connection header received
0926 22:52:38.500267 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.500268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.500277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.500322 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:38.500839 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.501368 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.502101 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.502296 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.502347 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.503333 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:38.503335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:38.503335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:38.503342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:38.504281 (+ 939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.504570 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.504571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.504572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.504623 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:38.504778 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:38.504779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:38.504779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:38.504907 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:38.504993 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.505113 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.505185 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:52:38.506529 30894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:52:38.508316 30894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:52:38.514043 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.510375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:36957 (local address 127.0.0.1:53524)
0926 22:52:38.510457 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.510488 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.510547 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.510873 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.510876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.510878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:38.510933 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.510935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.511928 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.511929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.512807 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.512809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.513722 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.513724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.513831 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.513845 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.513918 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.513968 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
I20260926 22:52:38.514050 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.510496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.13.148.1:36957)
0926 22:52:38.510709 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.510713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.510721 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:52:38.510752 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.510753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.510767 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.510840 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:38.511005 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.511848 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.512917 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.513642 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.513717 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.513813 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.513932 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.513969 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":81,"threads_started":1}
I20260926 22:52:38.514393 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.510192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45863 (local address 127.0.0.1:51406)
0926 22:52:38.510451 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.510488 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.510547 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.511121 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.511122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.511124 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:38.511174 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.511175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.512452 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.512453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.513346 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.513349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.514187 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.514188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.514247 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.514250 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.514294 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.514330 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":21,"thread_start_us":47,"threads_started":1}
I20260926 22:52:38.514432 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.510287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.13.148.2:45863)
0926 22:52:38.510982 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.510987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.510996 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:52:38.511033 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.511035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.511050 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.511117 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:38.511252 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.512345 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.513416 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.514032 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.514091 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.514149 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.514324 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.514364 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":75,"threads_started":1}
I20260926 22:52:38.514892 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.510213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45705 (local address 127.0.0.1:36104)
0926 22:52:38.510457 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.510490 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.510548 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.511387 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.511388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.511390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:38.511433 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.511434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.512603 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.512604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.513646 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.513648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.514677 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.514678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.514737 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.514740 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.514789 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.514826 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":63,"threads_started":1}
I20260926 22:52:38.514911 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.510301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.13.148.3:45705)
0926 22:52:38.511245 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.511249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.511257 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:52:38.511291 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.511293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.511306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.511377 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:38.511499 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.512512 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.513768 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.514452 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.514583 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.514641 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.514804 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.514843 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":68,"threads_started":1}
I20260926 22:52:38.514986 31019 tablet_service.cc:1467] Processing CreateTablet for tablet 28d103fb515645b799010dd550ad56f2 (DEFAULT_TABLE table=test-table [id=3c6742031a5243eb89301d3c54fd5740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:38.515075 31153 tablet_service.cc:1467] Processing CreateTablet for tablet 28d103fb515645b799010dd550ad56f2 (DEFAULT_TABLE table=test-table [id=3c6742031a5243eb89301d3c54fd5740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:38.515362 31153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d103fb515645b799010dd550ad56f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:38.515362 31019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d103fb515645b799010dd550ad56f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:38.515694 31287 tablet_service.cc:1467] Processing CreateTablet for tablet 28d103fb515645b799010dd550ad56f2 (DEFAULT_TABLE table=test-table [id=3c6742031a5243eb89301d3c54fd5740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:38.515925 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28d103fb515645b799010dd550ad56f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:38.518374 31367 tablet_bootstrap.cc:492] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: Bootstrap starting.
I20260926 22:52:38.518549 31368 tablet_bootstrap.cc:492] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544: Bootstrap starting.
I20260926 22:52:38.519205 31367 tablet_bootstrap.cc:654] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:38.519202 31368 tablet_bootstrap.cc:654] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:38.519770 31367 log.cc:824] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:38.519783 31368 log.cc:824] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:38.520087 31369 tablet_bootstrap.cc:492] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: Bootstrap starting.
I20260926 22:52:38.520505 31368 tablet_bootstrap.cc:492] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544: No bootstrap required, opened a new log
I20260926 22:52:38.520574 31368 ts_tablet_manager.cc:1397] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:52:38.520597 31367 tablet_bootstrap.cc:492] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: No bootstrap required, opened a new log
I20260926 22:52:38.520658 31367 ts_tablet_manager.cc:1397] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:38.520670 31369 tablet_bootstrap.cc:654] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:38.521216 31369 log.cc:824] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:38.522019 31369 tablet_bootstrap.cc:492] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: No bootstrap required, opened a new log
I20260926 22:52:38.522126 31369 ts_tablet_manager.cc:1397] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:38.522346 31367 raft_consensus.cc:348] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.522485 31367 raft_consensus.cc:374] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:38.522516 31367 raft_consensus.cc:729] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb7d3ca3655411c822d3095e1352534, State: Initialized, Role: FOLLOWER
I20260926 22:52:38.522652 31367 consensus_queue.cc:260] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.522621 31368 raft_consensus.cc:348] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.522809 31368 raft_consensus.cc:374] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:38.522856 31368 raft_consensus.cc:729] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d998f03bc46441f9dc3d6456f1bf544, State: Initialized, Role: FOLLOWER
I20260926 22:52:38.522914 31367 ts_tablet_manager.cc:1428] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:52:38.522964 31368 consensus_queue.cc:260] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.523067 31353 heartbeater.cc:503] Master 127.13.148.62:39995 was elected leader, sending a full tablet report...
I20260926 22:52:38.523303 31085 heartbeater.cc:503] Master 127.13.148.62:39995 was elected leader, sending a full tablet report...
I20260926 22:52:38.523296 31368 ts_tablet_manager.cc:1428] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:52:38.524093 31369 raft_consensus.cc:348] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.524255 31369 raft_consensus.cc:374] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:38.524303 31369 raft_consensus.cc:729] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf44920de0c246ee95c290199b1cb8b5, State: Initialized, Role: FOLLOWER
I20260926 22:52:38.524415 31369 consensus_queue.cc:260] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.524648 31369 ts_tablet_manager.cc:1428] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 22:52:38.524678 31219 heartbeater.cc:503] Master 127.13.148.62:39995 was elected leader, sending a full tablet report...
I20260926 22:52:38.535689 31375 raft_consensus.cc:482] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:38.535766 31375 raft_consensus.cc:504] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.536089 31375 leader_election.cc:288] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d998f03bc46441f9dc3d6456f1bf544 (127.13.148.1:36957), 2cb7d3ca3655411c822d3095e1352534 (127.13.148.3:45705)
I20260926 22:52:38.539654 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.536277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33639 (local address 127.13.148.1:36957)
0926 22:52:38.536332 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.536334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.536435 (+ 101us) server_negotiation.cc:407] Connection header received
0926 22:52:38.536480 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.536480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.536489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.536527 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:38.536640 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.537349 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.538785 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.539351 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.539414 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.539423 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.539558 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.539590 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 22:52:38.539654 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.536277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:42151 (local address 127.13.148.3:45705)
0926 22:52:38.536368 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.536370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.536376 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:52:38.536396 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.536397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.536403 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.536451 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:38.536601 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.537346 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.538792 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.539343 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.539414 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.539423 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.539559 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.539590 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 22:52:38.539664 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.536214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45705 (local address 127.13.148.2:42151)
0926 22:52:38.536271 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.536282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.536317 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.536464 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.536465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.536467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:38.536535 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.536537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.537414 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.537415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.538713 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.538716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.539415 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.539416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.539494 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.539496 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.539553 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.539594 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":60}
I20260926 22:52:38.539669 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.536214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:36957 (local address 127.13.148.2:33639)
0926 22:52:38.536364 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.536374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.536432 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.536531 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.536531 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.536532 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:38.536588 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.536589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.537414 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.537415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.538713 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.538716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.539415 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.539416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.539491 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.539495 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.539547 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.539586 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":89,"thread_start_us":83,"threads_started":1}
I20260926 22:52:38.539973 31307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d103fb515645b799010dd550ad56f2" candidate_uuid: "cf44920de0c246ee95c290199b1cb8b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb7d3ca3655411c822d3095e1352534" is_pre_election: true
I20260926 22:52:38.540117 31307 raft_consensus.cc:2454] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf44920de0c246ee95c290199b1cb8b5 in term 0.
I20260926 22:52:38.540203 31039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d103fb515645b799010dd550ad56f2" candidate_uuid: "cf44920de0c246ee95c290199b1cb8b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d998f03bc46441f9dc3d6456f1bf544" is_pre_election: true
I20260926 22:52:38.540321 31039 raft_consensus.cc:2454] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf44920de0c246ee95c290199b1cb8b5 in term 0.
I20260926 22:52:38.540503 31106 leader_election.cc:302] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb7d3ca3655411c822d3095e1352534, cf44920de0c246ee95c290199b1cb8b5; no voters:
I20260926 22:52:38.540618 31375 raft_consensus.cc:2781] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:52:38.540663 31375 raft_consensus.cc:482] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:52:38.540689 31375 raft_consensus.cc:3037] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:38.541306 31375 raft_consensus.cc:504] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.541405 31375 leader_election.cc:288] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [CANDIDATE]: Term 1 election: Requested vote from peers 8d998f03bc46441f9dc3d6456f1bf544 (127.13.148.1:36957), 2cb7d3ca3655411c822d3095e1352534 (127.13.148.3:45705)
I20260926 22:52:38.541528 31039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d103fb515645b799010dd550ad56f2" candidate_uuid: "cf44920de0c246ee95c290199b1cb8b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d998f03bc46441f9dc3d6456f1bf544"
I20260926 22:52:38.541538 31307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28d103fb515645b799010dd550ad56f2" candidate_uuid: "cf44920de0c246ee95c290199b1cb8b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb7d3ca3655411c822d3095e1352534"
I20260926 22:52:38.541596 31039 raft_consensus.cc:3037] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:38.541602 31307 raft_consensus.cc:3037] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:38.542280 31039 raft_consensus.cc:2454] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf44920de0c246ee95c290199b1cb8b5 in term 1.
I20260926 22:52:38.542316 31307 raft_consensus.cc:2454] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf44920de0c246ee95c290199b1cb8b5 in term 1.
I20260926 22:52:38.542444 31108 leader_election.cc:302] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d998f03bc46441f9dc3d6456f1bf544, cf44920de0c246ee95c290199b1cb8b5; no voters:
I20260926 22:52:38.542546 31375 raft_consensus.cc:2781] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:38.542683 31375 raft_consensus.cc:686] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 1 LEADER]: Becoming Leader. State: Replica: cf44920de0c246ee95c290199b1cb8b5, State: Running, Role: LEADER
I20260926 22:52:38.542765 31375 consensus_queue.cc:237] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } }
I20260926 22:52:38.543490 30894 catalog_manager.cc:5495] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf44920de0c246ee95c290199b1cb8b5 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "cf44920de0c246ee95c290199b1cb8b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf44920de0c246ee95c290199b1cb8b5" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 } health_report { overall_health: UNKNOWN } } }
I20260926 22:52:38.551587 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.549382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.13.148.2:45863)
0926 22:52:38.549416 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.549418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.549426 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:52:38.549497 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.549498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.549515 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.549574 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:52:38.549706 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.550179 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.550904 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.551103 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.551155 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.551294 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:52:38.551385 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:52:38.551449 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.551485 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.551504 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:52:38.553030 31307 raft_consensus.cc:1261] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 1 FOLLOWER]: Refusing update from remote peer cf44920de0c246ee95c290199b1cb8b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:38.553050 31039 raft_consensus.cc:1261] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 1 FOLLOWER]: Refusing update from remote peer cf44920de0c246ee95c290199b1cb8b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:38.553319 31375 consensus_queue.cc:1035] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d998f03bc46441f9dc3d6456f1bf544" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:38.553452 31377 consensus_queue.cc:1035] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb7d3ca3655411c822d3095e1352534" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:38.556304 31381 mvcc.cc:205] Tried to move back new op lower bound from 7333737097431007232 to 7333737097391730688. Current Snapshot: MvccSnapshot[applied={T|T < 7333737097431007232}]
I20260926 22:52:38.557984 30893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:38.558063 30893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:38.558729 30893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15058ea3b04a4fe6b17aa8d40e321bcc: Sending DeleteTablet for 3 replicas of tablet 28d103fb515645b799010dd550ad56f2
I20260926 22:52:38.559029 31287 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d103fb515645b799010dd550ad56f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:38 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:36104
I20260926 22:52:38.559078 31153 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d103fb515645b799010dd550ad56f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:38 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:51406
I20260926 22:52:38.559232 31389 tablet_replica.cc:330] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: stopping tablet replica
I20260926 22:52:38.559265 31019 tablet_service.cc:1514] Processing DeleteTablet for tablet 28d103fb515645b799010dd550ad56f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:38 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:53524
I20260926 22:52:38.559327 31389 raft_consensus.cc:2229] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:38.559429 31389 raft_consensus.cc:2258] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:38.559513 31391 tablet_replica.cc:330] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544: stopping tablet replica
I20260926 22:52:38.559624 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 30956
I20260926 22:52:38.559840 31391 raft_consensus.cc:2229] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:38.559896 31389 ts_tablet_manager.cc:1905] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:38.559935 31391 raft_consensus.cc:2258] T 28d103fb515645b799010dd550ad56f2 P 8d998f03bc46441f9dc3d6456f1bf544 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:38.561101 31390 tablet_replica.cc:330] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: stopping tablet replica
I20260926 22:52:38.561167 31390 raft_consensus.cc:2229] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:52:38.561241 31390 raft_consensus.cc:2258] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:38.561482 31389 ts_tablet_manager.cc:1918] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:38.561542 31389 log.cc:1196] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/28d103fb515645b799010dd550ad56f2
I20260926 22:52:38.561657 31390 ts_tablet_manager.cc:1905] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:38.561820 31389 ts_tablet_manager.cc:1939] T 28d103fb515645b799010dd550ad56f2 P 2cb7d3ca3655411c822d3095e1352534: Deleting consensus metadata
I20260926 22:52:38.562175 30878 catalog_manager.cc:4841] TS 2cb7d3ca3655411c822d3095e1352534 (127.13.148.3:45705): tablet 28d103fb515645b799010dd550ad56f2 (table test-table [id=3c6742031a5243eb89301d3c54fd5740]) successfully deleted
I20260926 22:52:38.563194 31390 ts_tablet_manager.cc:1918] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:38.563242 31390 log.cc:1196] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/28d103fb515645b799010dd550ad56f2
I20260926 22:52:38.563493 31390 ts_tablet_manager.cc:1939] T 28d103fb515645b799010dd550ad56f2 P cf44920de0c246ee95c290199b1cb8b5: Deleting consensus metadata
I20260926 22:52:38.563822 30880 catalog_manager.cc:4841] TS cf44920de0c246ee95c290199b1cb8b5 (127.13.148.2:45863): tablet 28d103fb515645b799010dd550ad56f2 (table test-table [id=3c6742031a5243eb89301d3c54fd5740]) successfully deleted
W20260926 22:52:38.566995 30880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:36957 (error 108)
I20260926 22:52:38.567195 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 31090
I20260926 22:52:38.567771 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.567290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:36957 (local address 127.0.0.1:53538)
0926 22:52:38.567710 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.567744 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:36957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
W20260926 22:52:38.567986 30880 catalog_manager.cc:4584] TS 8d998f03bc46441f9dc3d6456f1bf544 (127.13.148.1:36957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28d103fb515645b799010dd550ad56f2: Network error: Client connection negotiation failed: client connection to 127.13.148.1:36957: connect: Connection refused (error 111)
I20260926 22:52:38.571609 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 31224
I20260926 22:52:38.575286 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 30862
2026-09-26T22:52:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1744 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T22:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:38Z Disabled control of system clock
I20260926 22:52:38.599890 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46251
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:38.677698 31396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:38.677939 31396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:38.677959 31396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:38.679054 31396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:38.679087 31396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:38.679098 31396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:38.679109 31396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:38.679121 31396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:38.680364 31396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:38.680549 31396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:38.680778 31396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:38.685159 31401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:38.685138 31404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:38.685138 31402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.685163 31396 server_base.cc:1034] running on GCE node
I20260926 22:52:38.685608 31396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:38.685901 31396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:38.687042 31396 hybrid_clock.cc:648] HybridClock initialized: now 1790463158687029 us; error 29 us; skew 500 ppm
I20260926 22:52:38.691807 31396 webserver.cc:460] Webserver started at http://127.13.148.62:46207/ using document root <none> and password file <none>
I20260926 22:52:38.692036 31396 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:38.692085 31396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:38.692214 31396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:38.693238 31396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "103686d7866a4c61b5e031c753d1bf8c"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "049c67440b6d95f7402ce92f45867dc8"
server_key_iv: "3aa4fd49c760e2db7ac96beaa6510a8d"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.693572 31396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "103686d7866a4c61b5e031c753d1bf8c"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "049c67440b6d95f7402ce92f45867dc8"
server_key_iv: "3aa4fd49c760e2db7ac96beaa6510a8d"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.694779 31396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:38.695430 31410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.695591 31396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:38.695670 31396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "103686d7866a4c61b5e031c753d1bf8c"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "049c67440b6d95f7402ce92f45867dc8"
server_key_iv: "3aa4fd49c760e2db7ac96beaa6510a8d"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.695735 31396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:38.728634 31396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:38.728925 31396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:38.729063 31396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:38.732681 31396 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46251
I20260926 22:52:38.732714 31462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46251 every 8 connection(s)
I20260926 22:52:38.733008 31396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:38.733968 31463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:38.733964 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 31396
I20260926 22:52:38.734061 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:38.734272 13904 external_mini_cluster.cc:1428] Setting key 2eb64d6e2147bfdd6a06c3056fac57e2
I20260926 22:52:38.736876 31463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c: Bootstrap starting.
I20260926 22:52:38.737514 31463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:38.737839 31463 log.cc:824] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:38.738435 31463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c: No bootstrap required, opened a new log
I20260926 22:52:38.739446 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.734764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.13.148.62:46251)
0926 22:52:38.734944 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.734951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.734975 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:52:38.735423 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.735430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.735492 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.735604 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:38.735809 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.736659 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.737668 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.738143 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.738251 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.738869 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:38.738874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:38.738874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:38.739023 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:38.739055 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:38.739083 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:38.739133 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.739215 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.739292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260926 22:52:38.740375 31463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103686d7866a4c61b5e031c753d1bf8c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46251 } }
I20260926 22:52:38.740500 31463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:38.740530 31463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103686d7866a4c61b5e031c753d1bf8c, State: Initialized, Role: FOLLOWER
I20260926 22:52:38.740635 31463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103686d7866a4c61b5e031c753d1bf8c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46251 } }
I20260926 22:52:38.740697 31463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:38.740726 31463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:38.740772 31463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:38.741247 31463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103686d7866a4c61b5e031c753d1bf8c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46251 } }
I20260926 22:52:38.741333 31463 leader_election.cc:302] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 103686d7866a4c61b5e031c753d1bf8c; no voters:
I20260926 22:52:38.741482 31463 leader_election.cc:288] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:38.741525 31468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:38.741660 31468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [term 1 LEADER]: Becoming Leader. State: Replica: 103686d7866a4c61b5e031c753d1bf8c, State: Running, Role: LEADER
I20260926 22:52:38.741719 31463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:38.741770 31468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103686d7866a4c61b5e031c753d1bf8c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46251 } }
I20260926 22:52:38.742096 31469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "103686d7866a4c61b5e031c753d1bf8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103686d7866a4c61b5e031c753d1bf8c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46251 } } }
I20260926 22:52:38.742189 31469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:38.742113 31470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103686d7866a4c61b5e031c753d1bf8c. Latest consensus state: current_term: 1 leader_uuid: "103686d7866a4c61b5e031c753d1bf8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103686d7866a4c61b5e031c753d1bf8c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46251 } } }
I20260926 22:52:38.742305 31470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:38.742486 31475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:38.742976 31475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:38.744329 31475 catalog_manager.cc:1347] Generated new cluster ID: e3de2bd206424adbb8135b4238da2669
I20260926 22:52:38.744370 31475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:38.750764 31475 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:38.751216 31475 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:38.764910 31475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c: Generated new TSK 0
I20260926 22:52:38.765094 31475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:38.769809 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46251
--builtin_ntp_servers=127.13.148.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:38.848227 31487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:38.848436 31487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:38.848465 31487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:38.849591 31487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:38.849637 31487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:38.849670 31487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:52:38.851039 31487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:38.851271 31487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:38.851547 31487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:38.856233 31493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:38.856253 31492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:38.856252 31495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.856366 31487 server_base.cc:1034] running on GCE node
I20260926 22:52:38.856812 31487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:38.857055 31487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:38.858191 31487 hybrid_clock.cc:648] HybridClock initialized: now 1790463158858175 us; error 30 us; skew 500 ppm
I20260926 22:52:38.859369 31487 webserver.cc:460] Webserver started at http://127.13.148.1:38115/ using document root <none> and password file <none>
I20260926 22:52:38.859592 31487 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:38.859642 31487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:38.859767 31487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:38.860785 31487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "2a75da31e6974e1bace1061e9e941f06"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "2ff0c8a6e45a6c1328939a6190e9f9af"
server_key_iv: "5a3cf20cd1eb0879a8fce4fcd877b9df"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.861104 31487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "2a75da31e6974e1bace1061e9e941f06"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "2ff0c8a6e45a6c1328939a6190e9f9af"
server_key_iv: "5a3cf20cd1eb0879a8fce4fcd877b9df"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.862354 31487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:38.862973 31501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.863154 31487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:38.863234 31487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "2a75da31e6974e1bace1061e9e941f06"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "2ff0c8a6e45a6c1328939a6190e9f9af"
server_key_iv: "5a3cf20cd1eb0879a8fce4fcd877b9df"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.863296 31487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:38.871929 31487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:38.872189 31487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:38.872305 31487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:38.872493 31487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:38.872777 31487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:38.872808 31487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.872834 31487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:38.872859 31487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.878203 31487 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:45465
I20260926 22:52:38.878257 31614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:45465 every 8 connection(s)
I20260926 22:52:38.878535 31487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:52:38.883397 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.879184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:51187 (local address 127.13.148.62:46251)
0926 22:52:38.879241 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.879243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.879479 (+ 236us) server_negotiation.cc:407] Connection header received
0926 22:52:38.879983 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.879985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.880001 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.880056 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:38.880721 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.881217 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.882208 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.882394 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.882447 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.883156 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:38.883157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:38.883158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:38.883162 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:38.883177 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:38.883180 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:38.883221 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:38.883291 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 22:52:38.883319 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:52:38.883589 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.879186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46251 (local address 127.13.148.1:51187)
0926 22:52:38.879366 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.879400 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.879888 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.880138 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.880141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.880149 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:38.880637 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.880642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.881297 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.881298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.882132 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.882135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:38.882461 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:38.882462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:38.882581 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:38.883032 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:38.883032 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:38.883058 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:38.883080 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:38.883227 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:38.883232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:38.883235 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:38.883291 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 22:52:38.883388 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260926 22:52:38.884034 31615 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46251
I20260926 22:52:38.884119 31615 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:38.884255 31615 heartbeater.cc:511] Master 127.13.148.62:46251 requested a full tablet report, sending...
I20260926 22:52:38.884521 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 31487
I20260926 22:52:38.884641 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:52:38.884732 31427 ts_manager.cc:194] Registered new tserver with Master: 2a75da31e6974e1bace1061e9e941f06 (127.13.148.1:45465)
I20260926 22:52:38.884837 13904 external_mini_cluster.cc:1428] Setting key 05dae28cce70463902b9b04bbac3d385
I20260926 22:52:38.885421 31427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.1:51187
I20260926 22:52:38.885885 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46251
--builtin_ntp_servers=127.13.148.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:38.964090 31618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:38.964308 31618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:38.964330 31618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:38.965454 31618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:38.965583 31618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:38.965633 31618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:52:38.967090 31618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:38.967316 31618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:38.967579 31618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:38.972365 31623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:38.972379 31626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.972440 31618 server_base.cc:1034] running on GCE node
W20260926 22:52:38.972379 31624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:38.972779 31618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:38.973007 31618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:38.974148 31618 hybrid_clock.cc:648] HybridClock initialized: now 1790463158974132 us; error 28 us; skew 500 ppm
I20260926 22:52:38.975337 31618 webserver.cc:460] Webserver started at http://127.13.148.2:35377/ using document root <none> and password file <none>
I20260926 22:52:38.975559 31618 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:38.975610 31618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:38.975732 31618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:38.976765 31618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "ad7f0e50cea6419b8b1229bfe59b4111"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "22400e04349a91e75d52f0fa0876b1eb"
server_key_iv: "4dd13cfda95004bbda42836400ee19df"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.977092 31618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "ad7f0e50cea6419b8b1229bfe59b4111"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "22400e04349a91e75d52f0fa0876b1eb"
server_key_iv: "4dd13cfda95004bbda42836400ee19df"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.978327 31618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:38.978951 31632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.979115 31618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:38.979195 31618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "ad7f0e50cea6419b8b1229bfe59b4111"
format_stamp: "Formatted at 2026-09-26 22:52:38 on dist-test-slave-98hb"
server_key: "22400e04349a91e75d52f0fa0876b1eb"
server_key_iv: "4dd13cfda95004bbda42836400ee19df"
server_key_version: "encryptionkey@0"
I20260926 22:52:38.979257 31618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:38.990993 31618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:38.991263 31618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:38.991384 31618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:38.991597 31618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:38.991890 31618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:38.991922 31618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.991950 31618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:38.991974 31618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:38.997303 31618 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:45097
I20260926 22:52:38.997339 31745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:45097 every 8 connection(s)
I20260926 22:52:38.997653 31618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:52:39.000082 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 31618
I20260926 22:52:39.000195 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:52:39.000404 13904 external_mini_cluster.cc:1428] Setting key 086a242e1eb0bbcd7778dad0225c9bc1
I20260926 22:52:39.001312 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46251
--builtin_ntp_servers=127.13.148.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 22:52:39.003060 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.998314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:46721 (local address 127.13.148.62:46251)
0926 22:52:38.998362 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:38.998364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:38.998624 (+ 260us) server_negotiation.cc:407] Connection header received
0926 22:52:38.999131 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:38.999132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:38.999141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:38.999192 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:38.999842 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.000548 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.001675 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.001960 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.002026 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.002836 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:39.002837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:39.002838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:39.002842 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:39.002857 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:39.002860 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:39.002900 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.002979 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.003002 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:52:39.003309 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:38.998318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46251 (local address 127.13.148.2:46721)
0926 22:52:38.998511 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:38.998550 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:38.999015 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:38.999251 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:38.999254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:38.999262 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:38.999776 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:38.999781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.000652 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.000653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.001583 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.001586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.002029 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.002030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.002160 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.002683 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:39.002683 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:39.002722 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:39.002750 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:39.002915 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:39.002919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:39.002922 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:39.002984 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:52:39.003087 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260926 22:52:39.003728 31746 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46251
I20260926 22:52:39.003818 31746 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:39.003988 31746 heartbeater.cc:511] Master 127.13.148.62:46251 requested a full tablet report, sending...
I20260926 22:52:39.004341 31427 ts_manager.cc:194] Registered new tserver with Master: ad7f0e50cea6419b8b1229bfe59b4111 (127.13.148.2:45097)
I20260926 22:52:39.004691 31427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.2:46721
W20260926 22:52:39.081446 31749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:39.081705 31749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:39.081772 31749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:39.083021 31749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:39.083062 31749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:39.083094 31749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:52:39.084435 31749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:39.084664 31749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:39.084934 31749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:39.089557 31754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:39.089536 31757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:39.089581 31749 server_base.cc:1034] running on GCE node
W20260926 22:52:39.089546 31755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:39.090006 31749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:39.090206 31749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:39.091336 31749 hybrid_clock.cc:648] HybridClock initialized: now 1790463159091324 us; error 31 us; skew 500 ppm
I20260926 22:52:39.092483 31749 webserver.cc:460] Webserver started at http://127.13.148.3:43301/ using document root <none> and password file <none>
I20260926 22:52:39.092691 31749 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:39.092736 31749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:39.092849 31749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:39.093907 31749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "9501e465904f470db349f0b66ec89bd4"
format_stamp: "Formatted at 2026-09-26 22:52:39 on dist-test-slave-98hb"
server_key: "fd29271a6ac21111b8a032c6c17eb82e"
server_key_iv: "15666536a51c21f86c9b9ae1698755de"
server_key_version: "encryptionkey@0"
I20260926 22:52:39.094230 31749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "9501e465904f470db349f0b66ec89bd4"
format_stamp: "Formatted at 2026-09-26 22:52:39 on dist-test-slave-98hb"
server_key: "fd29271a6ac21111b8a032c6c17eb82e"
server_key_iv: "15666536a51c21f86c9b9ae1698755de"
server_key_version: "encryptionkey@0"
I20260926 22:52:39.095361 31749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:39.095973 31763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:39.096122 31749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:39.096210 31749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "9501e465904f470db349f0b66ec89bd4"
format_stamp: "Formatted at 2026-09-26 22:52:39 on dist-test-slave-98hb"
server_key: "fd29271a6ac21111b8a032c6c17eb82e"
server_key_iv: "15666536a51c21f86c9b9ae1698755de"
server_key_version: "encryptionkey@0"
I20260926 22:52:39.096271 31749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:39.108170 31749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:39.108450 31749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:39.108563 31749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:39.108769 31749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:39.109081 31749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:39.109114 31749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:39.109143 31749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:39.109174 31749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:39.114677 31749 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:36551
I20260926 22:52:39.114742 31876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:36551 every 8 connection(s)
I20260926 22:52:39.115043 31749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:52:39.116315 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 31749
I20260926 22:52:39.116413 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:52:39.116598 13904 external_mini_cluster.cc:1428] Setting key d7030d3040e83b3b928a18eceb549204
I20260926 22:52:39.120414 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.115773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:35523 (local address 127.13.148.62:46251)
0926 22:52:39.115838 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:39.115840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:39.116073 (+ 233us) server_negotiation.cc:407] Connection header received
0926 22:52:39.116666 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:39.116667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:39.116680 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:39.116734 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:39.117431 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.117982 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.118964 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.119153 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.119209 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.119997 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:39.119998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:39.119999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:39.120004 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:39.120018 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:39.120022 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:39.120064 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.120304 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.120334 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 22:52:39.120615 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.115735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46251 (local address 127.13.148.3:35523)
0926 22:52:39.115938 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:39.115973 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:39.116541 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:39.116827 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:39.116832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:39.116842 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:39.117338 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.117343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.118066 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.118067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.118874 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.118876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.119227 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.119228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.119362 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.119863 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:39.119863 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:39.119891 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:39.119913 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:39.120070 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:39.120074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:39.120077 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:39.120275 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 22:52:39.120386 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260926 22:52:39.121090 31877 heartbeater.cc:347] Connected to a master server at 127.13.148.62:46251
I20260926 22:52:39.121179 31877 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:39.121361 31877 heartbeater.cc:511] Master 127.13.148.62:46251 requested a full tablet report, sending...
I20260926 22:52:39.121783 31427 ts_manager.cc:194] Registered new tserver with Master: 9501e465904f470db349f0b66ec89bd4 (127.13.148.3:36551)
I20260926 22:52:39.122116 31427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.3:35523
I20260926 22:52:39.128077 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:52:39.132150 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.129386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.13.148.62:46251)
0926 22:52:39.129451 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:39.129453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:39.129507 (+ 54us) server_negotiation.cc:407] Connection header received
0926 22:52:39.129548 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:39.129549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:39.129557 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:39.129613 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:39.129781 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.130293 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.130982 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.131187 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.131238 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.131911 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:39.131912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:39.131912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:39.131916 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:39.131929 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:39.131933 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:39.131973 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.132036 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.132058 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 22:52:39.133774 31427 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:52:39.135347 31427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:52:39.141825 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.137372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45465 (local address 127.0.0.1:33222)
0926 22:52:39.137632 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:39.137674 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:39.137766 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:39.137937 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:39.137940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:39.137942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:39.138004 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.138006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.139270 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.139271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.140496 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.140498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.141445 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.141457 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.141543 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.141568 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 22:52:39.141657 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 22:52:39.141726 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":198,"threads_started":1}
I20260926 22:52:39.141824 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.137554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.13.148.1:45465)
0926 22:52:39.137769 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:39.137774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:39.137785 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:52:39.137841 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:39.137844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:39.137858 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:39.137927 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:39.138087 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.139195 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.140582 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.141336 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.141415 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.141517 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.141676 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.141717 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260926 22:52:39.142750 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.137372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:36551 (local address 127.0.0.1:34094)
0926 22:52:39.137632 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:39.137674 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:39.137759 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:39.138273 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:39.138274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:39.138276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:39.138329 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.138330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.139476 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.139477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.140503 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.140504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.142463 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.142464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.142555 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.142560 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 22:52:39.142611 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:52:39.142677 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":11,"thread_start_us":77,"threads_started":1}
I20260926 22:52:39.142802 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.137711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.13.148.2:45097)
0926 22:52:39.137916 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:39.137920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:39.138083 (+ 163us) server_negotiation.cc:407] Connection header received
0926 22:52:39.138131 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:39.138133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:39.138147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:39.138215 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:39.139476 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.140296 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.141473 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.142304 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.142362 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.142441 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.142654 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.142724 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260926 22:52:39.142980 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.137376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45097 (local address 127.0.0.1:46230)
0926 22:52:39.138020 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:39.138029 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:39.138078 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:39.138275 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:39.138275 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:39.138277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:39.138329 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.138330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.140362 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.140362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.141389 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.141392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.142467 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.142467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.142523 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.142527 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:39.142588 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 22:52:39.142901 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":269,"spinlock_wait_cycles":384,"thread_start_us":138,"threads_started":1}
I20260926 22:52:39.142992 31549 tablet_service.cc:1467] Processing CreateTablet for tablet 013ae279842b496e90c3e9e493b6ef8e (DEFAULT_TABLE table=test-table [id=bc6efddd8b7043a4bd0d437ef05cc3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:39.143399 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.137543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.13.148.3:36551)
0926 22:52:39.137894 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:39.137899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:39.137909 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:52:39.137952 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:39.137954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:39.137969 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:39.138215 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:39.138390 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.139401 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.141550 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.142281 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.142341 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.142430 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.143294 (+ 864us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.143338 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":73,"threads_started":1}
I20260926 22:52:39.143409 31549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013ae279842b496e90c3e9e493b6ef8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:39.143667 31680 tablet_service.cc:1467] Processing CreateTablet for tablet 013ae279842b496e90c3e9e493b6ef8e (DEFAULT_TABLE table=test-table [id=bc6efddd8b7043a4bd0d437ef05cc3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:39.143992 31680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013ae279842b496e90c3e9e493b6ef8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:39.146106 31811 tablet_service.cc:1467] Processing CreateTablet for tablet 013ae279842b496e90c3e9e493b6ef8e (DEFAULT_TABLE table=test-table [id=bc6efddd8b7043a4bd0d437ef05cc3fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:39.146413 31811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 013ae279842b496e90c3e9e493b6ef8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:39.147156 31892 tablet_bootstrap.cc:492] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: Bootstrap starting.
I20260926 22:52:39.147598 31891 tablet_bootstrap.cc:492] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06: Bootstrap starting.
I20260926 22:52:39.147722 31892 tablet_bootstrap.cc:654] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:39.147933 31892 log.cc:824] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:39.148224 31891 tablet_bootstrap.cc:654] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:39.148509 31891 log.cc:824] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:39.148535 31892 tablet_bootstrap.cc:492] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: No bootstrap required, opened a new log
I20260926 22:52:39.148581 31892 ts_tablet_manager.cc:1397] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 22:52:39.149580 31895 tablet_bootstrap.cc:492] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: Bootstrap starting.
I20260926 22:52:39.150254 31895 tablet_bootstrap.cc:654] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:39.150559 31895 log.cc:824] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:39.150909 31892 raft_consensus.cc:348] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.151062 31892 raft_consensus.cc:374] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:39.151105 31892 raft_consensus.cc:729] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7f0e50cea6419b8b1229bfe59b4111, State: Initialized, Role: FOLLOWER
I20260926 22:52:39.151227 31892 consensus_queue.cc:260] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.151315 31895 tablet_bootstrap.cc:492] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: No bootstrap required, opened a new log
I20260926 22:52:39.151371 31895 ts_tablet_manager.cc:1397] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:52:39.151502 31892 ts_tablet_manager.cc:1428] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:52:39.151540 31746 heartbeater.cc:503] Master 127.13.148.62:46251 was elected leader, sending a full tablet report...
I20260926 22:52:39.152073 31891 tablet_bootstrap.cc:492] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06: No bootstrap required, opened a new log
I20260926 22:52:39.152132 31891 ts_tablet_manager.cc:1397] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 22:52:39.153621 31895 raft_consensus.cc:348] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.153759 31895 raft_consensus.cc:374] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:39.153790 31895 raft_consensus.cc:729] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9501e465904f470db349f0b66ec89bd4, State: Initialized, Role: FOLLOWER
I20260926 22:52:39.153900 31895 consensus_queue.cc:260] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.154135 31895 ts_tablet_manager.cc:1428] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:52:39.154179 31877 heartbeater.cc:503] Master 127.13.148.62:46251 was elected leader, sending a full tablet report...
I20260926 22:52:39.154259 31891 raft_consensus.cc:348] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.154407 31891 raft_consensus.cc:374] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:39.154445 31891 raft_consensus.cc:729] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a75da31e6974e1bace1061e9e941f06, State: Initialized, Role: FOLLOWER
I20260926 22:52:39.154556 31891 consensus_queue.cc:260] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.154814 31891 ts_tablet_manager.cc:1428] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:52:39.154856 31615 heartbeater.cc:503] Master 127.13.148.62:46251 was elected leader, sending a full tablet report...
I20260926 22:52:39.173368 31898 raft_consensus.cc:482] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:39.173491 31898 raft_consensus.cc:504] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.173882 31898 leader_election.cc:288] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad7f0e50cea6419b8b1229bfe59b4111 (127.13.148.2:45097), 2a75da31e6974e1bace1061e9e941f06 (127.13.148.1:45465)
I20260926 22:52:39.176554 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.174105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50963 (local address 127.13.148.1:45465)
0926 22:52:39.174146 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:39.174148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:39.174278 (+ 130us) server_negotiation.cc:407] Connection header received
0926 22:52:39.174308 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:39.174310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:39.174321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:39.174372 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:39.174500 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.174961 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.175886 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.176258 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.176317 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.176325 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.176454 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.176489 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:52:39.176586 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.174088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:52967 (local address 127.13.148.2:45097)
0926 22:52:39.174130 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:39.174133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:39.174141 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:52:39.174171 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:39.174172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:39.174181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:39.174240 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:39.174395 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.174884 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.175866 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.176229 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.176289 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.176298 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.176471 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.176512 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:52:39.176594 31879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.174015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45097 (local address 127.13.148.3:52967)
0926 22:52:39.174056 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:39.174067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:39.174103 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:39.174256 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:39.174257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:39.174259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:39.174334 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.174335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.174954 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.174955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.175788 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.175790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.176296 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.176297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.176363 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.176367 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:39.176454 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 22:52:39.176504 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":27}
I20260926 22:52:39.176608 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.174015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45465 (local address 127.13.148.3:50963)
0926 22:52:39.174214 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:39.174221 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:39.174264 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:39.174376 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:39.174376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:39.174378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:39.174443 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.174445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.175026 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.175027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.175802 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:39.175804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.176328 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.176329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:39.176394 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.176397 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:39.176440 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:52:39.176480 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":46,"thread_start_us":112,"threads_started":1}
I20260926 22:52:39.176939 31569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013ae279842b496e90c3e9e493b6ef8e" candidate_uuid: "9501e465904f470db349f0b66ec89bd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a75da31e6974e1bace1061e9e941f06" is_pre_election: true
I20260926 22:52:39.177104 31569 raft_consensus.cc:2454] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9501e465904f470db349f0b66ec89bd4 in term 0.
I20260926 22:52:39.177090 31700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013ae279842b496e90c3e9e493b6ef8e" candidate_uuid: "9501e465904f470db349f0b66ec89bd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" is_pre_election: true
I20260926 22:52:39.177232 31700 raft_consensus.cc:2454] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9501e465904f470db349f0b66ec89bd4 in term 0.
I20260926 22:52:39.177350 31767 leader_election.cc:302] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a75da31e6974e1bace1061e9e941f06, 9501e465904f470db349f0b66ec89bd4; no voters:
I20260926 22:52:39.177474 31898 raft_consensus.cc:2781] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:52:39.177531 31898 raft_consensus.cc:482] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:52:39.177547 31898 raft_consensus.cc:3037] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:39.178282 31898 raft_consensus.cc:504] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.178403 31898 leader_election.cc:288] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [CANDIDATE]: Term 1 election: Requested vote from peers ad7f0e50cea6419b8b1229bfe59b4111 (127.13.148.2:45097), 2a75da31e6974e1bace1061e9e941f06 (127.13.148.1:45465)
I20260926 22:52:39.178537 31569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013ae279842b496e90c3e9e493b6ef8e" candidate_uuid: "9501e465904f470db349f0b66ec89bd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a75da31e6974e1bace1061e9e941f06"
I20260926 22:52:39.178609 31700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "013ae279842b496e90c3e9e493b6ef8e" candidate_uuid: "9501e465904f470db349f0b66ec89bd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7f0e50cea6419b8b1229bfe59b4111"
I20260926 22:52:39.178619 31569 raft_consensus.cc:3037] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:39.178679 31700 raft_consensus.cc:3037] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:39.179505 31569 raft_consensus.cc:2454] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9501e465904f470db349f0b66ec89bd4 in term 1.
I20260926 22:52:39.179505 31700 raft_consensus.cc:2454] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9501e465904f470db349f0b66ec89bd4 in term 1.
I20260926 22:52:39.179662 31767 leader_election.cc:302] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a75da31e6974e1bace1061e9e941f06, 9501e465904f470db349f0b66ec89bd4; no voters:
I20260926 22:52:39.179735 31898 raft_consensus.cc:2781] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:39.179870 31898 raft_consensus.cc:686] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 1 LEADER]: Becoming Leader. State: Replica: 9501e465904f470db349f0b66ec89bd4, State: Running, Role: LEADER
I20260926 22:52:39.179951 31898 consensus_queue.cc:237] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } }
I20260926 22:52:39.180751 31427 catalog_manager.cc:5495] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9501e465904f470db349f0b66ec89bd4 (127.13.148.3). New cstate: current_term: 1 leader_uuid: "9501e465904f470db349f0b66ec89bd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9501e465904f470db349f0b66ec89bd4" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 36551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 } health_report { overall_health: UNKNOWN } } }
I20260926 22:52:39.185809 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.183674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.13.148.3:36551)
0926 22:52:39.183723 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:39.183725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:39.183731 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:52:39.183760 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:39.183761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:39.183775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:39.183830 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:52:39.183944 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.184392 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.185105 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:39.185302 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:39.185353 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:39.185491 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:52:39.185589 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:52:39.185657 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:39.185683 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 22:52:39.185703 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 22:52:39.187436 31700 raft_consensus.cc:1261] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 1 FOLLOWER]: Refusing update from remote peer 9501e465904f470db349f0b66ec89bd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:39.187451 31569 raft_consensus.cc:1261] T 013ae279842b496e90c3e9e493b6ef8e P 2a75da31e6974e1bace1061e9e941f06 [term 1 FOLLOWER]: Refusing update from remote peer 9501e465904f470db349f0b66ec89bd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:39.187769 31898 consensus_queue.cc:1035] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a75da31e6974e1bace1061e9e941f06" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:39.187898 31901 consensus_queue.cc:1035] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad7f0e50cea6419b8b1229bfe59b4111" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:39.190800 31904 mvcc.cc:205] Tried to move back new op lower bound from 7333737100029444096 to 7333737100001837056. Current Snapshot: MvccSnapshot[applied={T|T < 7333737100029444096}]
I20260926 22:52:39.192710 31906 mvcc.cc:205] Tried to move back new op lower bound from 7333737100029444096 to 7333737100001837056. Current Snapshot: MvccSnapshot[applied={T|T < 7333737100029444096}]
I20260926 22:52:39.193148 31905 mvcc.cc:205] Tried to move back new op lower bound from 7333737100029444096 to 7333737100001837056. Current Snapshot: MvccSnapshot[applied={T|T < 7333737100029444096}]
I20260926 22:52:39.193674 31427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:39.193795 31427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:39.196300 31427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 103686d7866a4c61b5e031c753d1bf8c: Sending DeleteTablet for 3 replicas of tablet 013ae279842b496e90c3e9e493b6ef8e
I20260926 22:52:39.196643 31811 tablet_service.cc:1514] Processing DeleteTablet for tablet 013ae279842b496e90c3e9e493b6ef8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:39 UTC) from {username='slave'} at 127.0.0.1:34094
I20260926 22:52:39.197003 31680 tablet_service.cc:1514] Processing DeleteTablet for tablet 013ae279842b496e90c3e9e493b6ef8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:39 UTC) from {username='slave'} at 127.0.0.1:46230
I20260926 22:52:39.197050 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 31487
I20260926 22:52:39.198482 31913 tablet_replica.cc:330] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: stopping tablet replica
I20260926 22:52:39.198601 31913 raft_consensus.cc:2229] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:52:39.198699 31913 raft_consensus.cc:2258] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:39.199141 31913 ts_tablet_manager.cc:1905] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:39.200953 31913 ts_tablet_manager.cc:1918] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:39.201025 31913 log.cc:1196] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/013ae279842b496e90c3e9e493b6ef8e
I20260926 22:52:39.201256 31913 ts_tablet_manager.cc:1939] T 013ae279842b496e90c3e9e493b6ef8e P 9501e465904f470db349f0b66ec89bd4: Deleting consensus metadata
I20260926 22:52:39.201659 31412 catalog_manager.cc:4841] TS 9501e465904f470db349f0b66ec89bd4 (127.13.148.3:36551): tablet 013ae279842b496e90c3e9e493b6ef8e (table test-table [id=bc6efddd8b7043a4bd0d437ef05cc3fc]) successfully deleted
I20260926 22:52:39.201866 31914 tablet_replica.cc:330] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: stopping tablet replica
I20260926 22:52:39.201983 31914 raft_consensus.cc:2229] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:39.202093 31914 raft_consensus.cc:2258] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:39.202517 31914 ts_tablet_manager.cc:1905] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 22:52:39.203884 31414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:45465 (error 108)
I20260926 22:52:39.204030 31914 ts_tablet_manager.cc:1918] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:39.204186 31914 log.cc:1196] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/013ae279842b496e90c3e9e493b6ef8e
I20260926 22:52:39.204365 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.204241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45465 (local address 127.0.0.1:33228)
0926 22:52:39.204286 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:39.204332 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 22:52:39.204473 31914 ts_tablet_manager.cc:1939] T 013ae279842b496e90c3e9e493b6ef8e P ad7f0e50cea6419b8b1229bfe59b4111: Deleting consensus metadata
W20260926 22:52:39.204483 31414 catalog_manager.cc:4584] TS 2a75da31e6974e1bace1061e9e941f06 (127.13.148.1:45465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 013ae279842b496e90c3e9e493b6ef8e: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45465: connect: Connection refused (error 111)
I20260926 22:52:39.204669 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 31618
I20260926 22:52:39.209367 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 31749
I20260926 22:52:39.209378 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:39.209249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45097 (local address 127.0.0.1:46234)
0926 22:52:39.209300 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:39.209337 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.2:45097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:52:39.213523 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 31396
2026-09-26T22:52:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (638 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:39 dist-test-slave-98hb krb5kdc[31918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:39 dist-test-slave-98hb krb5kdc[31918](info): set up 2 sockets
Sep 26 22:52:39 dist-test-slave-98hb krb5kdc[31918](info): commencing operation
krb5kdc: starting...
W20260926 22:52:40.161639 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:52:40.271705 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34145
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:40.349852 31931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:40.350039 31931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:40.350059 31931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:40.351168 31931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:40.351207 31931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:40.351220 31931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:40.351231 31931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:40.351241 31931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:40.352643 31931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:34145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:40.352847 31931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:40.353058 31931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:40.357939 31936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:40.358058 31937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:40.357954 31939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:40.358441 31931 server_base.cc:1034] running on GCE node
I20260926 22:52:40.358672 31931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:40.358939 31931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:40.360082 31931 hybrid_clock.cc:648] HybridClock initialized: now 1790463160360067 us; error 34 us; skew 500 ppm
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:40.362321 31931 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:52:40.362715 31931 webserver.cc:460] Webserver started at http://127.13.148.62:35313/ using document root <none> and password file <none>
I20260926 22:52:40.362916 31931 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:40.362958 31931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:40.363107 31931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:40.364084 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "25b71ad60fbe489c876a75a767339fcb"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "944bd83feaf5c83835077d1a6d415451"
server_key_iv: "41e2eb68d3726e73352c02aa57271f28"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.364392 31931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "25b71ad60fbe489c876a75a767339fcb"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "944bd83feaf5c83835077d1a6d415451"
server_key_iv: "41e2eb68d3726e73352c02aa57271f28"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.365564 31931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:52:40.366268 31946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.366411 31931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.366482 31931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "25b71ad60fbe489c876a75a767339fcb"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "944bd83feaf5c83835077d1a6d415451"
server_key_iv: "41e2eb68d3726e73352c02aa57271f28"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.366546 31931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:40.379825 31931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:40.381366 31931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:40.381487 31931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:40.385092 31931 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:34145
I20260926 22:52:40.385121 31998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:34145 every 8 connection(s)
I20260926 22:52:40.385406 31931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:40.385681 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 31931
I20260926 22:52:40.385828 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:40.386051 13904 external_mini_cluster.cc:1428] Setting key be61f215c0dfe2121f2d5730476b7e7b
I20260926 22:52:40.386334 31999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:40.388837 31999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb: Bootstrap starting.
I20260926 22:52:40.389467 31999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:40.389778 31999 log.cc:824] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:40.390430 31999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb: No bootstrap required, opened a new log
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:40.392617 31999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b71ad60fbe489c876a75a767339fcb" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34145 } }
I20260926 22:52:40.392783 31999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:40.392825 31999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b71ad60fbe489c876a75a767339fcb, State: Initialized, Role: FOLLOWER
I20260926 22:52:40.392952 31999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b71ad60fbe489c876a75a767339fcb" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34145 } }
I20260926 22:52:40.393025 31999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:40.393054 31999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:40.393100 31999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:40.393678 31999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b71ad60fbe489c876a75a767339fcb" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34145 } }
I20260926 22:52:40.393797 31999 leader_election.cc:302] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25b71ad60fbe489c876a75a767339fcb; no voters:
I20260926 22:52:40.393941 31999 leader_election.cc:288] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:40.393982 32004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:40.394148 32004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [term 1 LEADER]: Becoming Leader. State: Replica: 25b71ad60fbe489c876a75a767339fcb, State: Running, Role: LEADER
I20260926 22:52:40.394182 31999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:40.394302 32004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b71ad60fbe489c876a75a767339fcb" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34145 } }
I20260926 22:52:40.395133 32006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25b71ad60fbe489c876a75a767339fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b71ad60fbe489c876a75a767339fcb" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34145 } } }
I20260926 22:52:40.395399 32005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25b71ad60fbe489c876a75a767339fcb. Latest consensus state: current_term: 1 leader_uuid: "25b71ad60fbe489c876a75a767339fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b71ad60fbe489c876a75a767339fcb" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34145 } } }
I20260926 22:52:40.395535 32006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:40.395769 32005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:40.396433 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.386490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.13.148.62:34145)
0926 22:52:40.386655 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:40.386664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:40.386688 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:52:40.387295 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:40.387301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:40.387345 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:40.387461 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:40.388040 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.388881 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.389981 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.390463 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.390570 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.392968 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:40.392971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:40.392972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:40.392981 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:40.394765 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.395099 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.395102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.395102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.395200 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.395412 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.395413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.395413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.395701 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:40.396155 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:40.396209 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 22:52:40.396300 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
W20260926 22:52:40.401861 32019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:52:40.401906 32019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:52:40.401958 32021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:40.402133 32021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:40.403424 32021 catalog_manager.cc:1347] Generated new cluster ID: d0eb047fb03f4a8c9003f87ede8242d7
I20260926 22:52:40.403476 32021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:40.426934 32021 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:40.427632 32021 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:40.435855 32021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb: Generated new TSK 0
I20260926 22:52:40.436028 32021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:52:40.464329 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34145
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:40.541139 32025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:40.541327 32025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:40.541349 32025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:40.542449 32025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:40.542487 32025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:40.542514 32025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:52:40.543877 32025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:40.544085 32025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:40.544289 32025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:40.548769 32030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:40.548787 32033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:40.548892 32031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:40.548880 32025 server_base.cc:1034] running on GCE node
I20260926 22:52:40.549204 32025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:40.549417 32025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:40.550549 32025 hybrid_clock.cc:648] HybridClock initialized: now 1790463160550534 us; error 29 us; skew 500 ppm
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:40.552789 32025 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:52:40.553233 32025 webserver.cc:460] Webserver started at http://127.13.148.1:43993/ using document root <none> and password file <none>
I20260926 22:52:40.553443 32025 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:40.553488 32025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:40.553604 32025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:40.554653 32025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "5f85bd69e1b44c37817819dca74e9326"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "5a1fe678b218d2c259fbe48136a501cd"
server_key_iv: "b5c523bf349ca52a759dedebd7e80079"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.554973 32025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "5f85bd69e1b44c37817819dca74e9326"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "5a1fe678b218d2c259fbe48136a501cd"
server_key_iv: "b5c523bf349ca52a759dedebd7e80079"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.556128 32025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:40.556831 32040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.556975 32025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:40.557049 32025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "5f85bd69e1b44c37817819dca74e9326"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "5a1fe678b218d2c259fbe48136a501cd"
server_key_iv: "b5c523bf349ca52a759dedebd7e80079"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.557107 32025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:40.595034 32025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:40.597190 32025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:40.597329 32025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:40.597536 32025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:40.597856 32025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:40.597891 32025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.597920 32025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:40.597942 32025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.603150 32025 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:39805
I20260926 22:52:40.603214 32153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:39805 every 8 connection(s)
I20260926 22:52:40.603487 32025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:40.608804 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 32025
I20260926 22:52:40.608906 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:52:40.609108 13904 external_mini_cluster.cc:1428] Setting key 7035cc529832f8e873d1ceab1c8f2be7
I20260926 22:52:40.611778 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.604122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:35433 (local address 127.13.148.62:34145)
0926 22:52:40.604187 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:40.604189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:40.604369 (+ 180us) server_negotiation.cc:407] Connection header received
0926 22:52:40.604830 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:40.604831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:40.604840 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:40.604897 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:40.605787 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.606291 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.607199 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.607379 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.607421 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.609261 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:40.609262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:40.609263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:40.609270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:40.610559 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.610909 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.610910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.610910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.610967 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.611212 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.611212 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.611213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.611319 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:40.611367 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:40.611654 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 22:52:40.611715 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 22:52:40.612025 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.604095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:34145 (local address 127.13.148.1:35433)
0926 22:52:40.604249 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:40.604291 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:40.604755 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:40.604950 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:40.604954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:40.605485 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:40.605675 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.605680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.606354 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.606355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.607139 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.607141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.607429 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.607430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.607522 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.607886 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:40.607894 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:40.609169 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:40.610638 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:40.610644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:40.610646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:40.610849 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:40.611020 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:40.611020 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:40.611020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:40.611149 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:40.611371 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:40.611375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:40.611523 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 22:52:40.611638 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 22:52:40.611812 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260926 22:52:40.612479 32154 heartbeater.cc:347] Connected to a master server at 127.13.148.62:34145
I20260926 22:52:40.612562 32154 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:40.612725 32154 heartbeater.cc:511] Master 127.13.148.62:34145 requested a full tablet report, sending...
I20260926 22:52:40.613180 31963 ts_manager.cc:194] Registered new tserver with Master: 5f85bd69e1b44c37817819dca74e9326 (127.13.148.1:39805)
I20260926 22:52:40.613780 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:35433
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:52:40.632105 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34145
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:40.709303 32159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:40.709483 32159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:40.709504 32159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:40.710601 32159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:40.710639 32159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:40.710664 32159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:52:40.711974 32159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:40.712136 32159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:40.712299 32159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:40.716768 32165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:40.716777 32164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:40.716974 32167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:40.716931 32159 server_base.cc:1034] running on GCE node
I20260926 22:52:40.717149 32159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:40.717336 32159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:40.718499 32159 hybrid_clock.cc:648] HybridClock initialized: now 1790463160718484 us; error 28 us; skew 500 ppm
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:40.720676 32159 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:52:40.721092 32159 webserver.cc:460] Webserver started at http://127.13.148.2:36315/ using document root <none> and password file <none>
I20260926 22:52:40.721275 32159 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:40.721316 32159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:40.721428 32159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:40.722420 32159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "2fad146af54f46a6ac0b178d34426106"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "7d6ce6f32dc071b5fc1c66d742ad23fe"
server_key_iv: "6b8e97c161d1819f34203115df934e16"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.722719 32159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "2fad146af54f46a6ac0b178d34426106"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "7d6ce6f32dc071b5fc1c66d742ad23fe"
server_key_iv: "6b8e97c161d1819f34203115df934e16"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.723861 32159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:52:40.724522 32174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.724663 32159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:40.724735 32159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "2fad146af54f46a6ac0b178d34426106"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "7d6ce6f32dc071b5fc1c66d742ad23fe"
server_key_iv: "6b8e97c161d1819f34203115df934e16"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.724794 32159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:40.754722 32159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:40.756466 32159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:40.756598 32159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:40.756811 32159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:40.757102 32159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:40.757133 32159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.757155 32159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:40.757179 32159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.762660 32159 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:41079
I20260926 22:52:40.762712 32287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:41079 every 8 connection(s)
I20260926 22:52:40.762996 32159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:52:40.766374 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 32159
I20260926 22:52:40.766492 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:52:40.766685 13904 external_mini_cluster.cc:1428] Setting key 5746ccd907ea5b9fd6364cfd688709d4
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:40.772248 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.763639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33953 (local address 127.13.148.62:34145)
0926 22:52:40.763722 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:40.763724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:40.763912 (+ 188us) server_negotiation.cc:407] Connection header received
0926 22:52:40.764410 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:40.764411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:40.764421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:40.764462 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:40.765341 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.765866 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.767074 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.767299 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.767370 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.769820 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:40.769822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:40.769822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:40.769829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:40.771061 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.771418 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.771419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.771419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.771470 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.771681 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.771682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.771683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.771786 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:40.771841 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:40.772085 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 22:52:40.772158 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 22:52:40.772424 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.763608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:34145 (local address 127.13.148.2:33953)
0926 22:52:40.763801 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:40.763842 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:40.764314 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:40.764516 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:40.764519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:40.765054 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:40.765251 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.765256 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.765925 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.765926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.767003 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.767005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.767381 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.767382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.767528 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.767978 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:40.767987 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:40.769715 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:40.771133 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:40.771139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:40.771141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:40.771354 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:40.771523 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:40.771524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:40.771524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:40.771623 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:40.771839 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:40.771845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:40.771974 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 22:52:40.772058 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 22:52:40.772215 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":113,"threads_started":1}
I20260926 22:52:40.772830 32288 heartbeater.cc:347] Connected to a master server at 127.13.148.62:34145
I20260926 22:52:40.772912 32288 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:40.773072 32288 heartbeater.cc:511] Master 127.13.148.62:34145 requested a full tablet report, sending...
I20260926 22:52:40.773420 31963 ts_manager.cc:194] Registered new tserver with Master: 2fad146af54f46a6ac0b178d34426106 (127.13.148.2:41079)
I20260926 22:52:40.773948 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:33953
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:52:40.790447 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34145
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:40.867121 32293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:40.867313 32293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:40.867334 32293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:40.868438 32293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:40.868475 32293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:40.868505 32293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:52:40.870007 32293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:34145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:40.870220 32293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:40.870416 32293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:40.874888 32298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:40.875046 32293 server_base.cc:1034] running on GCE node
W20260926 22:52:40.874907 32301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:40.874910 32299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:40.875349 32293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:40.875579 32293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:40.876730 32293 hybrid_clock.cc:648] HybridClock initialized: now 1790463160876712 us; error 26 us; skew 500 ppm
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:40.879037 32293 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:52:40.879446 32293 webserver.cc:460] Webserver started at http://127.13.148.3:43967/ using document root <none> and password file <none>
I20260926 22:52:40.879632 32293 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:40.879678 32293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:40.879801 32293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:40.880787 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "878369cd30d2411898326ad79316ab46"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "026f48b45041945777a195d0865895f0"
server_key_iv: "8be053626cb90915dee33acd6c4aa4a3"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.881155 32293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "878369cd30d2411898326ad79316ab46"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "026f48b45041945777a195d0865895f0"
server_key_iv: "8be053626cb90915dee33acd6c4aa4a3"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.882457 32293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 22:52:40.883113 32308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.883265 32293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.883344 32293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "878369cd30d2411898326ad79316ab46"
format_stamp: "Formatted at 2026-09-26 22:52:40 on dist-test-slave-98hb"
server_key: "026f48b45041945777a195d0865895f0"
server_key_iv: "8be053626cb90915dee33acd6c4aa4a3"
server_key_version: "encryptionkey@0"
I20260926 22:52:40.883399 32293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:40.901136 32293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:40.902832 32293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:40.902961 32293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:40.903175 32293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:40.903448 32293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:40.903481 32293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.903510 32293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:40.903527 32293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:40.908830 32293 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:45455
I20260926 22:52:40.908886 32421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:45455 every 8 connection(s)
I20260926 22:52:40.909165 32293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:52:40.914338 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 32293
I20260926 22:52:40.914431 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:52:40.914605 13904 external_mini_cluster.cc:1428] Setting key 2845629e7a6bbe7d5d8bbffaac72bfda
Sep 26 22:52:40 dist-test-slave-98hb krb5kdc[31918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463160, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:40.917650 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.909838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:52729 (local address 127.13.148.62:34145)
0926 22:52:40.909895 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:40.909897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:40.910089 (+ 192us) server_negotiation.cc:407] Connection header received
0926 22:52:40.910551 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:40.910552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:40.910561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:40.910614 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:40.911467 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.911949 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.912919 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.913093 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.913134 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.915148 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:40.915150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:40.915150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:40.915159 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:40.916458 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.916814 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.916815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.916816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.916871 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.917078 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.917078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.917079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.917190 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:40.917243 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:40.917491 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 22:52:40.917564 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 22:52:40.917843 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.909822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:34145 (local address 127.13.148.3:52729)
0926 22:52:40.909964 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:40.910015 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:40.910477 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:40.910668 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:40.910671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:40.911208 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:40.911399 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.911404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.912050 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.912051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.912853 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.912855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.913148 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.913149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.913245 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.913674 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:40.913682 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:40.915059 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:40.916532 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:40.916538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:40.916540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:40.916749 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:40.916924 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:40.916925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:40.916925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:40.917026 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:40.917244 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:40.917250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:40.917379 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 22:52:40.917465 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 22:52:40.917614 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260926 22:52:40.918259 32422 heartbeater.cc:347] Connected to a master server at 127.13.148.62:34145
I20260926 22:52:40.918339 32422 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:40.918497 32422 heartbeater.cc:511] Master 127.13.148.62:34145 requested a full tablet report, sending...
I20260926 22:52:40.918813 31963 ts_manager.cc:194] Registered new tserver with Master: 878369cd30d2411898326ad79316ab46 (127.13.148.3:45455)
I20260926 22:52:40.919155 31963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:52729
I20260926 22:52:40.926189 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:52:40.932807 32002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.927410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.13.148.62:34145)
0926 22:52:40.927459 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:40.927461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:40.927550 (+ 89us) server_negotiation.cc:407] Connection header received
0926 22:52:40.927605 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:40.927606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:40.927613 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:40.927656 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:40.928124 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.928633 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.929280 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.929451 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.929493 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.930707 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:40.930708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:40.930708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:40.930714 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:40.931855 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.932184 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.932184 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.932185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.932229 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:40.932389 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:40.932390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:40.932390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:40.932500 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:40.932570 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:40.932677 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 22:52:40.932722 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:52:40.934162 31963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:52:40.935789 31963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:52:40.941970 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.938257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.13.148.1:39805)
0926 22:52:40.938414 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:40.938418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:40.938918 (+ 500us) server_negotiation.cc:407] Connection header received
0926 22:52:40.938964 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:40.938966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:40.938981 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:40.939050 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:40.939160 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.940130 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.941082 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.941571 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.941633 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.941690 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:40.941858 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 22:52:40.941896 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260926 22:52:40.941962 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.938054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:39805 (local address 127.0.0.1:49898)
0926 22:52:40.938746 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:40.938756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:40.938809 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:40.939050 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:40.939051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:40.939053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:40.939100 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.939101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.940214 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.940215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.941011 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.941014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.941644 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.941645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.941756 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.941772 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 22:52:40.941837 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 22:52:40.941887 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":8,"thread_start_us":64,"threads_started":1}
I20260926 22:52:40.942162 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.938016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45455 (local address 127.0.0.1:60142)
0926 22:52:40.938118 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:40.938158 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:40.938220 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:40.938567 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:40.938570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:40.938574 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:40.938653 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.938655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.940021 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.940022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.941047 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.941048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.941958 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.941959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.942021 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.942025 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:40.942054 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 22:52:40.942092 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260926 22:52:40.942441 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.938018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41079 (local address 127.0.0.1:43478)
0926 22:52:40.938354 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:40.938364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:40.938405 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:40.938642 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:40.938642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:40.938644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:40.938681 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.938682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.939670 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.939671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.941037 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:40.941039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.942010 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.942011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:40.942084 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.942087 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:40.942130 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:52:40.942379 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":2,"thread_start_us":41,"threads_started":1}
I20260926 22:52:40.942535 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.938213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.13.148.3:45455)
0926 22:52:40.938392 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:40.938396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:40.938406 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:52:40.938444 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:40.938446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:40.938461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:40.938527 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:40.938830 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.939908 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.941123 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.941864 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.942329 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.942392 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:40.942416 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 22:52:40.942454 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260926 22:52:40.942720 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:40.938189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.13.148.2:41079)
0926 22:52:40.938488 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:40.938490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:40.938498 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:52:40.938534 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:40.938536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:40.938552 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:40.938640 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:40.938746 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.939584 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.941123 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:40.941865 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:40.941930 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:40.941983 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:40.942603 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 22:52:40.942649 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":82,"threads_started":1}
I20260926 22:52:40.942773 32088 tablet_service.cc:1467] Processing CreateTablet for tablet fc0cc92959fc4bc6b4271ddea580e727 (DEFAULT_TABLE table=test-table [id=deb3163584ad4293bb6ba7c280168237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:40.943171 32088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0cc92959fc4bc6b4271ddea580e727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:40.943156 32356 tablet_service.cc:1467] Processing CreateTablet for tablet fc0cc92959fc4bc6b4271ddea580e727 (DEFAULT_TABLE table=test-table [id=deb3163584ad4293bb6ba7c280168237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:40.943441 32356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0cc92959fc4bc6b4271ddea580e727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:40.943370 32222 tablet_service.cc:1467] Processing CreateTablet for tablet fc0cc92959fc4bc6b4271ddea580e727 (DEFAULT_TABLE table=test-table [id=deb3163584ad4293bb6ba7c280168237]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:40.943645 32222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0cc92959fc4bc6b4271ddea580e727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:40.945770 32439 tablet_bootstrap.cc:492] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Bootstrap starting.
I20260926 22:52:40.946444 32439 tablet_bootstrap.cc:654] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:40.946806 32440 tablet_bootstrap.cc:492] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326: Bootstrap starting.
I20260926 22:52:40.946806 32441 tablet_bootstrap.cc:492] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Bootstrap starting.
I20260926 22:52:40.946969 32439 log.cc:824] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:40.947410 32440 tablet_bootstrap.cc:654] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:40.947410 32441 tablet_bootstrap.cc:654] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:40.947665 32439 tablet_bootstrap.cc:492] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: No bootstrap required, opened a new log
I20260926 22:52:40.947724 32439 ts_tablet_manager.cc:1397] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:40.947940 32440 log.cc:824] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:40.947940 32441 log.cc:824] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:40.948603 32441 tablet_bootstrap.cc:492] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: No bootstrap required, opened a new log
I20260926 22:52:40.948603 32440 tablet_bootstrap.cc:492] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326: No bootstrap required, opened a new log
I20260926 22:52:40.948680 32440 ts_tablet_manager.cc:1397] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:40.948791 32441 ts_tablet_manager.cc:1397] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:52:40.949299 32439 raft_consensus.cc:348] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:40.949424 32439 raft_consensus.cc:374] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:40.949443 32439 raft_consensus.cc:729] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878369cd30d2411898326ad79316ab46, State: Initialized, Role: FOLLOWER
I20260926 22:52:40.949532 32439 consensus_queue.cc:260] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:40.949785 32439 ts_tablet_manager.cc:1428] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:52:40.949815 32422 heartbeater.cc:503] Master 127.13.148.62:34145 was elected leader, sending a full tablet report...
I20260926 22:52:40.950551 32440 raft_consensus.cc:348] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:40.950716 32440 raft_consensus.cc:374] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:40.950747 32440 raft_consensus.cc:729] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f85bd69e1b44c37817819dca74e9326, State: Initialized, Role: FOLLOWER
I20260926 22:52:40.950851 32440 consensus_queue.cc:260] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:40.950923 32441 raft_consensus.cc:348] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:40.951071 32441 raft_consensus.cc:374] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:40.951102 32440 ts_tablet_manager.cc:1428] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:52:40.951102 32441 raft_consensus.cc:729] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fad146af54f46a6ac0b178d34426106, State: Initialized, Role: FOLLOWER
I20260926 22:52:40.951123 32154 heartbeater.cc:503] Master 127.13.148.62:34145 was elected leader, sending a full tablet report...
I20260926 22:52:40.951215 32441 consensus_queue.cc:260] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:40.951460 32441 ts_tablet_manager.cc:1428] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:52:40.951489 32288 heartbeater.cc:503] Master 127.13.148.62:34145 was elected leader, sending a full tablet report...
W20260926 22:52:41.013562 32289 tablet.cc:2369] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 22:52:41.104151 32155 tablet.cc:2369] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:52:41.120682 32446 raft_consensus.cc:482] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:41.120802 32446 raft_consensus.cc:504] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:41.121177 32446 leader_election.cc:288] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fad146af54f46a6ac0b178d34426106 (127.13.148.2:41079), 878369cd30d2411898326ad79316ab46 (127.13.148.3:45455)
I20260926 22:52:41.124080 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.121678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:54635 (local address 127.13.148.3:45455)
0926 22:52:41.121720 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.121722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.121748 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:52:41.121770 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.121771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.121779 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.121827 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:41.121992 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.122441 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.123403 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.123775 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.123836 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.123844 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.123978 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.124014 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:52:41.124081 32436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.121506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:59013 (local address 127.13.148.2:41079)
0926 22:52:41.121549 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.121551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.121560 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:52:41.121588 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.121590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.121598 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.121653 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:41.121834 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.122380 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.123397 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.123765 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.123823 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.123831 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.123973 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.124006 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:52:41.124104 32449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.121336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:41079 (local address 127.13.148.1:59013)
0926 22:52:41.121460 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.121474 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.121531 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.121667 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.121668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.121670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:41.121767 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.121769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.122453 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.122454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.123309 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.123312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.123835 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.123836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.123910 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.123914 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.123961 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.124003 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":12,"thread_start_us":60,"threads_started":1}
I20260926 22:52:41.124109 32448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.121351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:45455 (local address 127.13.148.1:54635)
0926 22:52:41.121481 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.121489 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.121551 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.121846 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.121847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.121850 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:41.121922 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.121925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.122519 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.122520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.123311 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.123313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.123848 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.123849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.123912 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.123915 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.123962 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.124006 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260926 22:52:41.124431 32242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0cc92959fc4bc6b4271ddea580e727" candidate_uuid: "5f85bd69e1b44c37817819dca74e9326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fad146af54f46a6ac0b178d34426106" is_pre_election: true
I20260926 22:52:41.124598 32242 raft_consensus.cc:2454] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f85bd69e1b44c37817819dca74e9326 in term 0.
I20260926 22:52:41.124745 32376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0cc92959fc4bc6b4271ddea580e727" candidate_uuid: "5f85bd69e1b44c37817819dca74e9326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878369cd30d2411898326ad79316ab46" is_pre_election: true
I20260926 22:52:41.124862 32042 leader_election.cc:302] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fad146af54f46a6ac0b178d34426106, 5f85bd69e1b44c37817819dca74e9326; no voters:
I20260926 22:52:41.124897 32376 raft_consensus.cc:2454] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f85bd69e1b44c37817819dca74e9326 in term 0.
I20260926 22:52:41.124985 32446 raft_consensus.cc:2781] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:52:41.125077 32446 raft_consensus.cc:482] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:52:41.125101 32446 raft_consensus.cc:3037] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:41.125820 32446 raft_consensus.cc:504] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:41.125948 32446 leader_election.cc:288] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [CANDIDATE]: Term 1 election: Requested vote from peers 2fad146af54f46a6ac0b178d34426106 (127.13.148.2:41079), 878369cd30d2411898326ad79316ab46 (127.13.148.3:45455)
I20260926 22:52:41.126085 32376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0cc92959fc4bc6b4271ddea580e727" candidate_uuid: "5f85bd69e1b44c37817819dca74e9326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878369cd30d2411898326ad79316ab46"
I20260926 22:52:41.126117 32242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0cc92959fc4bc6b4271ddea580e727" candidate_uuid: "5f85bd69e1b44c37817819dca74e9326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fad146af54f46a6ac0b178d34426106"
I20260926 22:52:41.126155 32376 raft_consensus.cc:3037] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:41.126188 32242 raft_consensus.cc:3037] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:41.126947 32242 raft_consensus.cc:2454] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f85bd69e1b44c37817819dca74e9326 in term 1.
I20260926 22:52:41.126950 32376 raft_consensus.cc:2454] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f85bd69e1b44c37817819dca74e9326 in term 1.
I20260926 22:52:41.127116 32043 leader_election.cc:302] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f85bd69e1b44c37817819dca74e9326, 878369cd30d2411898326ad79316ab46; no voters:
I20260926 22:52:41.127228 32446 raft_consensus.cc:2781] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:41.127364 32446 raft_consensus.cc:686] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 1 LEADER]: Becoming Leader. State: Replica: 5f85bd69e1b44c37817819dca74e9326, State: Running, Role: LEADER
I20260926 22:52:41.127439 32446 consensus_queue.cc:237] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } }
I20260926 22:52:41.128291 31963 catalog_manager.cc:5495] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f85bd69e1b44c37817819dca74e9326 (127.13.148.1). New cstate: current_term: 1 leader_uuid: "5f85bd69e1b44c37817819dca74e9326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f85bd69e1b44c37817819dca74e9326" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 39805 } health_report { overall_health: HEALTHY } } }
W20260926 22:52:41.159660 32423 tablet.cc:2369] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:52:41.169418 32438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.167169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.13.148.1:39805)
0926 22:52:41.167224 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.167226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.167233 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:52:41.167268 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.167269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.167290 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.167356 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:52:41.167467 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.167938 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.168686 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.168878 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.168937 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.169084 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:52:41.169201 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:52:41.169278 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.169332 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.169341 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:52:41.171183 32376 raft_consensus.cc:1261] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 1 FOLLOWER]: Refusing update from remote peer 5f85bd69e1b44c37817819dca74e9326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:41.171185 32242 raft_consensus.cc:1261] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 1 FOLLOWER]: Refusing update from remote peer 5f85bd69e1b44c37817819dca74e9326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:41.171561 32450 consensus_queue.cc:1035] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fad146af54f46a6ac0b178d34426106" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:41.171689 32446 consensus_queue.cc:1035] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "878369cd30d2411898326ad79316ab46" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:41.173848 32453 mvcc.cc:205] Tried to move back new op lower bound from 7333737108154851328 to 7333737107978715136. Current Snapshot: MvccSnapshot[applied={T|T < 7333737108154851328}]
I20260926 22:52:41.174083 32455 mvcc.cc:205] Tried to move back new op lower bound from 7333737108154851328 to 7333737107978715136. Current Snapshot: MvccSnapshot[applied={T|T < 7333737108154851328}]
I20260926 22:52:41.174522 32454 mvcc.cc:205] Tried to move back new op lower bound from 7333737108154851328 to 7333737107978715136. Current Snapshot: MvccSnapshot[applied={T|T < 7333737108154851328 or (T in {7333737108154851328})}]
I20260926 22:52:41.176396 31963 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:41.176496 31963 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:41.177175 31963 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25b71ad60fbe489c876a75a767339fcb: Sending DeleteTablet for 3 replicas of tablet fc0cc92959fc4bc6b4271ddea580e727
I20260926 22:52:41.177470 32222 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0cc92959fc4bc6b4271ddea580e727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:41 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:43478
I20260926 22:52:41.177474 32088 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0cc92959fc4bc6b4271ddea580e727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:41 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:49898
I20260926 22:52:41.177503 32356 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0cc92959fc4bc6b4271ddea580e727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:41 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:60142
I20260926 22:52:41.177700 32464 tablet_replica.cc:330] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: stopping tablet replica
I20260926 22:52:41.177853 32463 tablet_replica.cc:330] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326: stopping tablet replica
I20260926 22:52:41.177951 32463 raft_consensus.cc:2229] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:52:41.177855 32464 raft_consensus.cc:2229] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:41.178064 32463 raft_consensus.cc:2258] T fc0cc92959fc4bc6b4271ddea580e727 P 5f85bd69e1b44c37817819dca74e9326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:41.178149 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 32025
I20260926 22:52:41.178201 32465 tablet_replica.cc:330] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: stopping tablet replica
I20260926 22:52:41.178277 32465 raft_consensus.cc:2229] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:41.178351 32465 raft_consensus.cc:2258] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:41.178113 32464 raft_consensus.cc:2258] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:41.179060 32464 ts_tablet_manager.cc:1905] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:41.179718 32465 ts_tablet_manager.cc:1905] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:41.180467 32464 ts_tablet_manager.cc:1918] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:41.180523 32464 log.cc:1196] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/fc0cc92959fc4bc6b4271ddea580e727
I20260926 22:52:41.180716 32464 ts_tablet_manager.cc:1939] T fc0cc92959fc4bc6b4271ddea580e727 P 2fad146af54f46a6ac0b178d34426106: Deleting consensus metadata
I20260926 22:52:41.181185 32465 ts_tablet_manager.cc:1918] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:41.181253 32465 log.cc:1196] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/fc0cc92959fc4bc6b4271ddea580e727
I20260926 22:52:41.181429 31948 catalog_manager.cc:4841] TS 2fad146af54f46a6ac0b178d34426106 (127.13.148.2:41079): tablet fc0cc92959fc4bc6b4271ddea580e727 (table test-table [id=deb3163584ad4293bb6ba7c280168237]) successfully deleted
I20260926 22:52:41.181545 32465 ts_tablet_manager.cc:1939] T fc0cc92959fc4bc6b4271ddea580e727 P 878369cd30d2411898326ad79316ab46: Deleting consensus metadata
I20260926 22:52:41.181931 31949 catalog_manager.cc:4841] TS 878369cd30d2411898326ad79316ab46 (127.13.148.3:45455): tablet fc0cc92959fc4bc6b4271ddea580e727 (table test-table [id=deb3163584ad4293bb6ba7c280168237]) successfully deleted
W20260926 22:52:41.183769 31950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:39805 (error 108)
I20260926 22:52:41.183979 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 32159
I20260926 22:52:41.184561 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.184105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:39805 (local address 127.0.0.1:49906)
0926 22:52:41.184498 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.184536 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372}
W20260926 22:52:41.184681 31950 catalog_manager.cc:4584] TS 5f85bd69e1b44c37817819dca74e9326 (127.13.148.1:39805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc0cc92959fc4bc6b4271ddea580e727: Network error: Client connection negotiation failed: client connection to 127.13.148.1:39805: connect: Connection refused (error 111)
I20260926 22:52:41.188060 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 32293
I20260926 22:52:41.194339 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 31931
2026-09-26T22:52:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1981 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T22:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:41Z Disabled control of system clock
I20260926 22:52:41.220306 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:32945
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:37895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:32945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:41.296944 32470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:41.297148 32470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:41.297168 32470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:41.298247 32470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:41.298280 32470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:41.298291 32470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:41.298302 32470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:41.298312 32470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:41.299546 32470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:32945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:32945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:41.299718 32470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:41.299947 32470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:41.304220 32475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:41.304324 32478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:41.304220 32476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:41.304425 32470 server_base.cc:1034] running on GCE node
I20260926 22:52:41.304653 32470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:41.304909 32470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:41.306038 32470 hybrid_clock.cc:648] HybridClock initialized: now 1790463161306024 us; error 26 us; skew 500 ppm
I20260926 22:52:41.307140 32470 webserver.cc:460] Webserver started at http://127.13.148.62:37197/ using document root <none> and password file <none>
I20260926 22:52:41.307358 32470 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:41.307404 32470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:41.307525 32470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:41.308480 32470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "34119cd914bf4b4195e0bd145b68bf13"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "ca77fbc4ebc2dfe36d4f55ad1472d814"
server_key_iv: "e75481cd608662aa40ad9388bd07096c"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.308780 32470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "34119cd914bf4b4195e0bd145b68bf13"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "ca77fbc4ebc2dfe36d4f55ad1472d814"
server_key_iv: "e75481cd608662aa40ad9388bd07096c"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.309960 32470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:41.310637 32484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.310787 32470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:41.310865 32470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "34119cd914bf4b4195e0bd145b68bf13"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "ca77fbc4ebc2dfe36d4f55ad1472d814"
server_key_iv: "e75481cd608662aa40ad9388bd07096c"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.310925 32470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:41.323529 32470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:41.323802 32470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:41.323927 32470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:41.327827 32470 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:32945
I20260926 22:52:41.327884 32536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:32945 every 8 connection(s)
I20260926 22:52:41.328181 32470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:41.329047 32537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:41.331213 32537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13: Bootstrap starting.
I20260926 22:52:41.331738 32537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:41.331984 32537 log.cc:824] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:41.332578 32537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13: No bootstrap required, opened a new log
I20260926 22:52:41.333880 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 32470
I20260926 22:52:41.333952 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:41.334122 13904 external_mini_cluster.cc:1428] Setting key e05dd1eec1e8f5c947657f873e58f23e
I20260926 22:52:41.334144 32537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34119cd914bf4b4195e0bd145b68bf13" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32945 } }
I20260926 22:52:41.334301 32537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:41.334342 32537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34119cd914bf4b4195e0bd145b68bf13, State: Initialized, Role: FOLLOWER
I20260926 22:52:41.334482 32537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34119cd914bf4b4195e0bd145b68bf13" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32945 } }
I20260926 22:52:41.334578 32537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:41.334607 32537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:41.334652 32537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:41.335119 32537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34119cd914bf4b4195e0bd145b68bf13" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32945 } }
I20260926 22:52:41.335207 32537 leader_election.cc:302] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34119cd914bf4b4195e0bd145b68bf13; no voters:
I20260926 22:52:41.335361 32537 leader_election.cc:288] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:41.335388 32542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:41.335503 32542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [term 1 LEADER]: Becoming Leader. State: Replica: 34119cd914bf4b4195e0bd145b68bf13, State: Running, Role: LEADER
I20260926 22:52:41.335610 32537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:41.335623 32542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34119cd914bf4b4195e0bd145b68bf13" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32945 } }
I20260926 22:52:41.336042 32543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34119cd914bf4b4195e0bd145b68bf13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34119cd914bf4b4195e0bd145b68bf13" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32945 } } }
I20260926 22:52:41.336038 32544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34119cd914bf4b4195e0bd145b68bf13. Latest consensus state: current_term: 1 leader_uuid: "34119cd914bf4b4195e0bd145b68bf13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34119cd914bf4b4195e0bd145b68bf13" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 32945 } } }
I20260926 22:52:41.336130 32543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:41.336134 32544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:41.336414 32550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:41.336887 32550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:41.338337 32550 catalog_manager.cc:1347] Generated new cluster ID: 57c57766a05743efafbb12332d6471af
I20260926 22:52:41.338390 32550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:41.340199 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.334542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.13.148.62:32945)
0926 22:52:41.334746 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.334756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.334805 (+ 49us) server_negotiation.cc:407] Connection header received
0926 22:52:41.334944 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.334949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.335008 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.335140 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:41.335338 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.337033 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.337991 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.338466 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.339262 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.339406 (+ 144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:41.339409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:41.339409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:41.339597 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:41.339630 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:41.339642 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:41.339905 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.339933 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.340005 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260926 22:52:41.354416 32550 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:41.355110 32550 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:41.361951 32550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13: Generated new TSK 0
I20260926 22:52:41.362128 32550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:41.364439 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32945
--builtin_ntp_servers=127.13.148.20:37895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:41.441318 32561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:41.441504 32561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:41.441531 32561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:41.442632 32561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:41.442670 32561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:41.442701 32561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:52:41.443970 32561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:41.444190 32561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:41.444443 32561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:41.449996 32569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:41.449903 32566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:41.449903 32567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:41.450098 32561 server_base.cc:1034] running on GCE node
I20260926 22:52:41.450419 32561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:41.450639 32561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:41.451781 32561 hybrid_clock.cc:648] HybridClock initialized: now 1790463161451763 us; error 34 us; skew 500 ppm
I20260926 22:52:41.452872 32561 webserver.cc:460] Webserver started at http://127.13.148.1:34309/ using document root <none> and password file <none>
I20260926 22:52:41.453080 32561 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:41.453127 32561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:41.453313 32561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:41.454355 32561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "e8ec59022b0246ac9d362ba8d25ceaaf"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "0e1399cc5000a112b08ab9cbb1a3b8bb"
server_key_iv: "2d05f23617ca61859b673a3347a4693d"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.454674 32561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "e8ec59022b0246ac9d362ba8d25ceaaf"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "0e1399cc5000a112b08ab9cbb1a3b8bb"
server_key_iv: "2d05f23617ca61859b673a3347a4693d"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.455821 32561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:41.456470 32575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.456625 32561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:41.456707 32561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "e8ec59022b0246ac9d362ba8d25ceaaf"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "0e1399cc5000a112b08ab9cbb1a3b8bb"
server_key_iv: "2d05f23617ca61859b673a3347a4693d"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.456768 32561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:41.471112 32561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:41.471369 32561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:41.471488 32561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:41.471680 32561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:41.471968 32561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:41.472004 32561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.472031 32561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:41.472054 32561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.477205 32561 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:41397
I20260926 22:52:41.477258 32688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:41397 every 8 connection(s)
I20260926 22:52:41.477536 32561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:52:41.478971 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 32561
I20260926 22:52:41.479065 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:52:41.479238 13904 external_mini_cluster.cc:1428] Setting key 2439b3e67a2a8b389aa093e19b899291
I20260926 22:52:41.480031 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32945
--builtin_ntp_servers=127.13.148.20:37895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 22:52:41.482589 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.478188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:47031 (local address 127.13.148.62:32945)
0926 22:52:41.478239 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.478241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.478443 (+ 202us) server_negotiation.cc:407] Connection header received
0926 22:52:41.478956 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.478956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.478969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.479018 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:41.479727 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.480269 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.481314 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.481508 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.481560 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.482376 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:41.482377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:41.482377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:41.482381 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:41.482394 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:41.482397 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:41.482432 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.482508 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.482534 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:52:41.482803 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.478162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32945 (local address 127.13.148.1:47031)
0926 22:52:41.478339 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.478369 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.478866 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.479117 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.479122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.479132 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:41.479646 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.479651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.480350 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.480351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.481243 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.481245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.481580 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.481581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.481724 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.482231 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:41.482232 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:41.482268 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:41.482293 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:41.482442 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:41.482445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:41.482448 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.482510 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.482608 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260926 22:52:41.483256 32689 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32945
I20260926 22:52:41.483346 32689 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:41.483528 32689 heartbeater.cc:511] Master 127.13.148.62:32945 requested a full tablet report, sending...
I20260926 22:52:41.484045 32501 ts_manager.cc:194] Registered new tserver with Master: e8ec59022b0246ac9d362ba8d25ceaaf (127.13.148.1:41397)
I20260926 22:52:41.484896 32501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.1:47031
W20260926 22:52:41.559928 32692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:41.560113 32692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:41.560139 32692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:41.561264 32692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:41.561302 32692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:41.561334 32692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:52:41.562660 32692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:41.562878 32692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:41.563135 32692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:41.567617 32697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:41.567744 32698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:41.567775 32692 server_base.cc:1034] running on GCE node
W20260926 22:52:41.567628 32700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:41.568027 32692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:41.568240 32692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:41.569370 32692 hybrid_clock.cc:648] HybridClock initialized: now 1790463161569360 us; error 28 us; skew 500 ppm
I20260926 22:52:41.570533 32692 webserver.cc:460] Webserver started at http://127.13.148.2:36883/ using document root <none> and password file <none>
I20260926 22:52:41.570745 32692 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:41.570788 32692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:41.570901 32692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:41.571869 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "7d2243cf32d144a5a70edb4ea7162198"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "d27bd13e33edf6d43d7de366c2ad61b6"
server_key_iv: "4d6464f869ae3c72a2e6deb9c999b851"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.572175 32692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "7d2243cf32d144a5a70edb4ea7162198"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "d27bd13e33edf6d43d7de366c2ad61b6"
server_key_iv: "4d6464f869ae3c72a2e6deb9c999b851"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.573310 32692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:52:41.574007 32706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.574184 32692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 22:52:41.574252 32692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "7d2243cf32d144a5a70edb4ea7162198"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "d27bd13e33edf6d43d7de366c2ad61b6"
server_key_iv: "4d6464f869ae3c72a2e6deb9c999b851"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.574317 32692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:41.581286 32692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:41.581543 32692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:41.581664 32692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:41.581880 32692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:41.582183 32692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:41.582216 32692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.582242 32692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:41.582267 32692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.587524 32692 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:45599
I20260926 22:52:41.587569 351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:45599 every 8 connection(s)
I20260926 22:52:41.587859 32692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:52:41.592801 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.588535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:53367 (local address 127.13.148.62:32945)
0926 22:52:41.588595 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.588598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.588793 (+ 195us) server_negotiation.cc:407] Connection header received
0926 22:52:41.589278 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.589279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.589289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.589341 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:41.590007 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.590492 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.591484 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.591689 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.591743 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.592528 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:41.592529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:41.592529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:41.592533 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:41.592553 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:41.592557 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:41.592614 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.592720 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.592744 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:52:41.592983 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.588510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32945 (local address 127.13.148.2:53367)
0926 22:52:41.588684 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.588716 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.589197 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.589392 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.589395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.589403 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:41.589922 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.589927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.590601 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.590602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.591410 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.591412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.591760 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.591761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.591870 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.592372 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:41.592373 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:41.592401 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:41.592423 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:41.592643 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:41.592649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:41.592651 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.592719 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.592797 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260926 22:52:41.593428 352 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32945
I20260926 22:52:41.593508 352 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:41.593672 352 heartbeater.cc:511] Master 127.13.148.62:32945 requested a full tablet report, sending...
I20260926 22:52:41.594089 32501 ts_manager.cc:194] Registered new tserver with Master: 7d2243cf32d144a5a70edb4ea7162198 (127.13.148.2:45599)
I20260926 22:52:41.594424 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 32692
I20260926 22:52:41.594437 32501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.2:53367
I20260926 22:52:41.594519 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:52:41.594709 13904 external_mini_cluster.cc:1428] Setting key f851fb1419c7dcfe1757c94ce8874b9c
I20260926 22:52:41.595595 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32945
--builtin_ntp_servers=127.13.148.20:37895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:41.672125 355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:41.672299 355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:41.672318 355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:41.673385 355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:41.673422 355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:41.673448 355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:52:41.674739 355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:37895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:32945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:41.674934 355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:41.675185 355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:41.679690 363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:41.679708 360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:41.679711 361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:41.679930 355 server_base.cc:1034] running on GCE node
I20260926 22:52:41.680137 355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:41.680322 355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:41.681471 355 hybrid_clock.cc:648] HybridClock initialized: now 1790463161681462 us; error 25 us; skew 500 ppm
I20260926 22:52:41.682621 355 webserver.cc:460] Webserver started at http://127.13.148.3:44521/ using document root <none> and password file <none>
I20260926 22:52:41.682839 355 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:41.682883 355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:41.682997 355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:41.683969 355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "a8aed9c97fb64af79886c639aac5cb6b"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "67f13017561abbc57613a317f010ad09"
server_key_iv: "64967d726c470191d5c310e8e514f56b"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.684293 355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "a8aed9c97fb64af79886c639aac5cb6b"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "67f13017561abbc57613a317f010ad09"
server_key_iv: "64967d726c470191d5c310e8e514f56b"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.685389 355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:41.686012 369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.686142 355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.686211 355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "a8aed9c97fb64af79886c639aac5cb6b"
format_stamp: "Formatted at 2026-09-26 22:52:41 on dist-test-slave-98hb"
server_key: "67f13017561abbc57613a317f010ad09"
server_key_iv: "64967d726c470191d5c310e8e514f56b"
server_key_version: "encryptionkey@0"
I20260926 22:52:41.686271 355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:41.702265 355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:41.702523 355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:41.702636 355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:41.702832 355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:41.703135 355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:41.703167 355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.703188 355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:41.703212 355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:41.708525 355 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:35183
I20260926 22:52:41.708598 482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:35183 every 8 connection(s)
I20260926 22:52:41.708868 355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:52:41.709770 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 355
I20260926 22:52:41.709877 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:52:41.710060 13904 external_mini_cluster.cc:1428] Setting key 4ddb1a3d7c3091ef5c39893dda3a8723
I20260926 22:52:41.713824 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.709522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:60851 (local address 127.13.148.62:32945)
0926 22:52:41.709569 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.709571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.709845 (+ 274us) server_negotiation.cc:407] Connection header received
0926 22:52:41.710409 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.710410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.710421 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.710462 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:41.711109 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.711608 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.712542 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.712756 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.712811 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.713556 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:41.713557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:41.713558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:41.713562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:41.713584 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:41.713589 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:41.713628 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.713698 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.713745 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:52:41.714021 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.709520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:32945 (local address 127.13.148.3:60851)
0926 22:52:41.709699 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.709757 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.710315 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.710531 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.710535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.710545 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:41.711019 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.711023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.711689 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.711690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.712464 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.712466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.712823 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.712824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.712960 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.713421 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:41.713421 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:41.713457 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:41.713479 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:41.713634 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:41.713638 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:41.713641 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.713701 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.713822 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260926 22:52:41.714440 483 heartbeater.cc:347] Connected to a master server at 127.13.148.62:32945
I20260926 22:52:41.714521 483 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:41.714695 483 heartbeater.cc:511] Master 127.13.148.62:32945 requested a full tablet report, sending...
I20260926 22:52:41.715010 32501 ts_manager.cc:194] Registered new tserver with Master: a8aed9c97fb64af79886c639aac5cb6b (127.13.148.3:35183)
I20260926 22:52:41.715370 32501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.3:60851
I20260926 22:52:41.721534 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:52:41.725227 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.722796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51950 (local address 127.13.148.62:32945)
0926 22:52:41.722843 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.722845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.722941 (+ 96us) server_negotiation.cc:407] Connection header received
0926 22:52:41.722990 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.722991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.722999 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.723033 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:41.723153 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.723586 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.724260 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.724427 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.724467 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.725016 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:41.725017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:41.725017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:41.725021 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:41.725033 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:41.725037 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:41.725107 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.725147 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.725156 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:52:41.726658 32501 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:52:41.728160 32501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:52:41.733721 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.730075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.13.148.3:35183)
0926 22:52:41.730317 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.730322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.730359 (+ 37us) server_negotiation.cc:407] Connection header received
0926 22:52:41.730447 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.730449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.730463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.730548 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:41.730694 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.731752 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.732724 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.733279 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.733339 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.733418 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.733606 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.733643 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260926 22:52:41.733723 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.730004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:41397 (local address 127.0.0.1:52064)
0926 22:52:41.730317 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.730327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.730375 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.730656 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.730657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.730658 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:41.730701 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.730703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.731639 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.731640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.732634 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.732636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.733437 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.733438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.733511 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.733523 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.733582 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.733645 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260926 22:52:41.733983 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.730124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.13.148.1:41397)
0926 22:52:41.730506 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.730509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.730517 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:52:41.730556 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.730558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.730572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.730650 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:41.730782 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.731566 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.732723 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.733371 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.733427 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.733509 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.733881 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.733922 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":82,"threads_started":1}
I20260926 22:52:41.733753 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.730004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:35183 (local address 127.0.0.1:58720)
0926 22:52:41.730156 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.730305 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.730375 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.730555 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.730557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.730559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:41.730617 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.730619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.731837 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.731838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.732638 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.732641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.733452 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.733453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.733530 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.733533 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.733587 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.733650 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":23,"thread_start_us":54,"threads_started":1}
I20260926 22:52:41.734292 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.730314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.13.148.2:45599)
0926 22:52:41.730489 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.730493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.730782 (+ 289us) server_negotiation.cc:407] Connection header received
0926 22:52:41.730825 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.730827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.730842 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.730907 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:41.731062 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.732068 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.733247 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.733959 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.734023 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.734094 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.734183 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.734216 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260926 22:52:41.734480 495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.730199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45599 (local address 127.0.0.1:55122)
0926 22:52:41.730400 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.730590 (+ 190us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.730642 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.730969 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.730970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.730972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:41.731003 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.731004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.732140 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.732141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.733170 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.733172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.734033 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.734034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.734105 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.734109 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.734171 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.734207 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260926 22:52:41.734666 32623 tablet_service.cc:1467] Processing CreateTablet for tablet a921ca9b8cac4d9caa509dc130a3daad (DEFAULT_TABLE table=test-table [id=357c81b3b7f34d7d96148464d9bb1ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:41.735054 32623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a921ca9b8cac4d9caa509dc130a3daad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:41.734993 417 tablet_service.cc:1467] Processing CreateTablet for tablet a921ca9b8cac4d9caa509dc130a3daad (DEFAULT_TABLE table=test-table [id=357c81b3b7f34d7d96148464d9bb1ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:41.735092 32754 tablet_service.cc:1467] Processing CreateTablet for tablet a921ca9b8cac4d9caa509dc130a3daad (DEFAULT_TABLE table=test-table [id=357c81b3b7f34d7d96148464d9bb1ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:41.735316 417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a921ca9b8cac4d9caa509dc130a3daad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:41.735363 32754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a921ca9b8cac4d9caa509dc130a3daad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:41.737636 497 tablet_bootstrap.cc:492] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf: Bootstrap starting.
I20260926 22:52:41.738229 497 tablet_bootstrap.cc:654] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:41.738389 499 tablet_bootstrap.cc:492] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Bootstrap starting.
I20260926 22:52:41.738767 497 log.cc:824] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:41.738987 499 tablet_bootstrap.cc:654] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:41.739282 499 log.cc:824] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:41.739476 497 tablet_bootstrap.cc:492] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf: No bootstrap required, opened a new log
I20260926 22:52:41.739544 497 ts_tablet_manager.cc:1397] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 22:52:41.739599 498 tablet_bootstrap.cc:492] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Bootstrap starting.
I20260926 22:52:41.740186 498 tablet_bootstrap.cc:654] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:41.740484 498 log.cc:824] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:41.740969 499 tablet_bootstrap.cc:492] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: No bootstrap required, opened a new log
I20260926 22:52:41.741037 499 ts_tablet_manager.cc:1397] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:52:41.741991 498 tablet_bootstrap.cc:492] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: No bootstrap required, opened a new log
I20260926 22:52:41.742048 498 ts_tablet_manager.cc:1397] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:52:41.742045 497 raft_consensus.cc:348] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.742211 497 raft_consensus.cc:374] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:41.742280 497 raft_consensus.cc:729] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ec59022b0246ac9d362ba8d25ceaaf, State: Initialized, Role: FOLLOWER
I20260926 22:52:41.742421 497 consensus_queue.cc:260] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.742684 497 ts_tablet_manager.cc:1428] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 22:52:41.742816 32689 heartbeater.cc:503] Master 127.13.148.62:32945 was elected leader, sending a full tablet report...
I20260926 22:52:41.743397 499 raft_consensus.cc:348] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.743580 499 raft_consensus.cc:374] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:41.743629 499 raft_consensus.cc:729] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8aed9c97fb64af79886c639aac5cb6b, State: Initialized, Role: FOLLOWER
I20260926 22:52:41.743758 499 consensus_queue.cc:260] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.744016 499 ts_tablet_manager.cc:1428] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:52:41.744043 483 heartbeater.cc:503] Master 127.13.148.62:32945 was elected leader, sending a full tablet report...
I20260926 22:52:41.744297 498 raft_consensus.cc:348] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.744419 498 raft_consensus.cc:374] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:41.744448 498 raft_consensus.cc:729] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2243cf32d144a5a70edb4ea7162198, State: Initialized, Role: FOLLOWER
I20260926 22:52:41.744566 498 consensus_queue.cc:260] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.744817 498 ts_tablet_manager.cc:1428] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 22:52:41.744932 352 heartbeater.cc:503] Master 127.13.148.62:32945 was elected leader, sending a full tablet report...
W20260926 22:52:41.838341 353 tablet.cc:2369] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:52:41.945106 503 raft_consensus.cc:482] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:41.945227 503 raft_consensus.cc:504] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.945561 503 leader_election.cc:288] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8aed9c97fb64af79886c639aac5cb6b (127.13.148.3:35183), 7d2243cf32d144a5a70edb4ea7162198 (127.13.148.2:45599)
I20260926 22:52:41.948244 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.945817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:53501 (local address 127.13.148.3:35183)
0926 22:52:41.945861 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.945863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.945875 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:52:41.945907 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.945908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.945917 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.945983 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:41.946138 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.946659 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.947577 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.947937 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.947995 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.948003 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.948133 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.948180 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 22:52:41.948246 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.945723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:35183 (local address 127.13.148.1:53501)
0926 22:52:41.945786 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.945798 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.945842 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.945996 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.945997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.945999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:41.946062 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.946074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.946727 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.946728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.947495 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.947498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.948005 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.948006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.948076 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.948078 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.948121 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.948165 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 22:52:41.948261 506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.945842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:45599 (local address 127.13.148.1:48035)
0926 22:52:41.946005 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.946015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:41.946066 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:41.946164 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:41.946164 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:41.946166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:41.946227 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.946229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.946815 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.946816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.947518 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:41.947520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.948019 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.948020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:41.948084 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.948087 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:41.948134 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 22:52:41.948169 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260926 22:52:41.948246 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.945928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:48035 (local address 127.13.148.2:45599)
0926 22:52:41.945972 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.945974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.946068 (+ 94us) server_negotiation.cc:407] Connection header received
0926 22:52:41.946104 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.946105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.946114 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.946161 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:41.946281 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.946745 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.947600 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.947941 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.947999 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.948007 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.948145 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.948181 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 22:52:41.948590 306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a921ca9b8cac4d9caa509dc130a3daad" candidate_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d2243cf32d144a5a70edb4ea7162198" is_pre_election: true
I20260926 22:52:41.948591 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a921ca9b8cac4d9caa509dc130a3daad" candidate_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8aed9c97fb64af79886c639aac5cb6b" is_pre_election: true
I20260926 22:52:41.948740 306 raft_consensus.cc:2454] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ec59022b0246ac9d362ba8d25ceaaf in term 0.
I20260926 22:52:41.948740 437 raft_consensus.cc:2454] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ec59022b0246ac9d362ba8d25ceaaf in term 0.
I20260926 22:52:41.948956 32579 leader_election.cc:302] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8aed9c97fb64af79886c639aac5cb6b, e8ec59022b0246ac9d362ba8d25ceaaf; no voters:
I20260926 22:52:41.949069 503 raft_consensus.cc:2781] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:52:41.949131 503 raft_consensus.cc:482] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:52:41.949154 503 raft_consensus.cc:3037] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:41.949863 503 raft_consensus.cc:504] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.949970 503 leader_election.cc:288] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [CANDIDATE]: Term 1 election: Requested vote from peers a8aed9c97fb64af79886c639aac5cb6b (127.13.148.3:35183), 7d2243cf32d144a5a70edb4ea7162198 (127.13.148.2:45599)
I20260926 22:52:41.950112 437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a921ca9b8cac4d9caa509dc130a3daad" candidate_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8aed9c97fb64af79886c639aac5cb6b"
I20260926 22:52:41.950119 306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a921ca9b8cac4d9caa509dc130a3daad" candidate_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d2243cf32d144a5a70edb4ea7162198"
I20260926 22:52:41.950186 306 raft_consensus.cc:3037] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:41.950186 437 raft_consensus.cc:3037] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:41.951035 437 raft_consensus.cc:2454] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ec59022b0246ac9d362ba8d25ceaaf in term 1.
I20260926 22:52:41.951035 306 raft_consensus.cc:2454] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ec59022b0246ac9d362ba8d25ceaaf in term 1.
I20260926 22:52:41.951182 32579 leader_election.cc:302] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8aed9c97fb64af79886c639aac5cb6b, e8ec59022b0246ac9d362ba8d25ceaaf; no voters:
I20260926 22:52:41.951272 503 raft_consensus.cc:2781] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:41.951421 503 raft_consensus.cc:686] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 1 LEADER]: Becoming Leader. State: Replica: e8ec59022b0246ac9d362ba8d25ceaaf, State: Running, Role: LEADER
I20260926 22:52:41.951527 503 consensus_queue.cc:237] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } }
I20260926 22:52:41.952307 32501 catalog_manager.cc:5495] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf reported cstate change: term changed from 0 to 1, leader changed from <none> to e8ec59022b0246ac9d362ba8d25ceaaf (127.13.148.1). New cstate: current_term: 1 leader_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ec59022b0246ac9d362ba8d25ceaaf" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 41397 } health_report { overall_health: HEALTHY } } }
W20260926 22:52:41.959340 484 tablet.cc:2369] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:52:41.960747 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.958450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.13.148.1:41397)
0926 22:52:41.958498 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:41.958500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:41.958508 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:52:41.958566 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:41.958567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:41.958586 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:41.958647 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:52:41.958751 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.959205 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.960028 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:41.960219 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:41.960268 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:41.960406 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:52:41.960512 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:52:41.960593 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:41.960659 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 22:52:41.960667 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:52:41.962211 306 raft_consensus.cc:1261] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 1 FOLLOWER]: Refusing update from remote peer e8ec59022b0246ac9d362ba8d25ceaaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:41.962242 437 raft_consensus.cc:1261] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 1 FOLLOWER]: Refusing update from remote peer e8ec59022b0246ac9d362ba8d25ceaaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:41.962518 503 consensus_queue.cc:1035] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8aed9c97fb64af79886c639aac5cb6b" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 35183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:41.962652 507 consensus_queue.cc:1035] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2243cf32d144a5a70edb4ea7162198" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:41.965476 511 mvcc.cc:205] Tried to move back new op lower bound from 7333737111395102720 to 7333737111353901056. Current Snapshot: MvccSnapshot[applied={T|T < 7333737111395102720 or (T in {7333737111395102720})}]
I20260926 22:52:41.967132 32497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:41.967224 32497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:41.967917 32497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34119cd914bf4b4195e0bd145b68bf13: Sending DeleteTablet for 3 replicas of tablet a921ca9b8cac4d9caa509dc130a3daad
I20260926 22:52:41.968242 32623 tablet_service.cc:1514] Processing DeleteTablet for tablet a921ca9b8cac4d9caa509dc130a3daad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:41 UTC) from {username='slave'} at 127.0.0.1:52064
I20260926 22:52:41.968241 32754 tablet_service.cc:1514] Processing DeleteTablet for tablet a921ca9b8cac4d9caa509dc130a3daad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:41 UTC) from {username='slave'} at 127.0.0.1:55122
I20260926 22:52:41.968266 417 tablet_service.cc:1514] Processing DeleteTablet for tablet a921ca9b8cac4d9caa509dc130a3daad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:41 UTC) from {username='slave'} at 127.0.0.1:58720
I20260926 22:52:41.968441 520 tablet_replica.cc:330] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf: stopping tablet replica
I20260926 22:52:41.968474 521 tablet_replica.cc:330] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: stopping tablet replica
I20260926 22:52:41.968541 520 raft_consensus.cc:2229] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:52:41.968552 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 32561
I20260926 22:52:41.968634 521 raft_consensus.cc:2229] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:41.968689 520 raft_consensus.cc:2258] T a921ca9b8cac4d9caa509dc130a3daad P e8ec59022b0246ac9d362ba8d25ceaaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:41.968717 521 raft_consensus.cc:2258] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:41.969499 521 ts_tablet_manager.cc:1905] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:41.971122 521 ts_tablet_manager.cc:1918] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:41.971182 521 log.cc:1196] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/a921ca9b8cac4d9caa509dc130a3daad
I20260926 22:52:41.971374 521 ts_tablet_manager.cc:1939] T a921ca9b8cac4d9caa509dc130a3daad P 7d2243cf32d144a5a70edb4ea7162198: Deleting consensus metadata
I20260926 22:52:41.971709 32488 catalog_manager.cc:4841] TS 7d2243cf32d144a5a70edb4ea7162198 (127.13.148.2:45599): tablet a921ca9b8cac4d9caa509dc130a3daad (table test-table [id=357c81b3b7f34d7d96148464d9bb1ac3]) successfully deleted
W20260926 22:52:41.973381 32487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:41397 (error 108)
I20260926 22:52:41.973845 522 tablet_replica.cc:330] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: stopping tablet replica
I20260926 22:52:41.973857 495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:41.973700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:41397 (local address 127.0.0.1:52078)
0926 22:52:41.973782 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:41.973824 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 22:52:41.973858 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 32692
I20260926 22:52:41.973958 522 raft_consensus.cc:2229] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:41.974093 522 raft_consensus.cc:2258] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 22:52:41.974534 32487 catalog_manager.cc:4584] TS e8ec59022b0246ac9d362ba8d25ceaaf (127.13.148.1:41397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a921ca9b8cac4d9caa509dc130a3daad: Network error: Client connection negotiation failed: client connection to 127.13.148.1:41397: connect: Connection refused (error 111)
I20260926 22:52:41.975477 522 ts_tablet_manager.cc:1905] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:41.976963 522 ts_tablet_manager.cc:1918] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:41.977030 522 log.cc:1196] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/a921ca9b8cac4d9caa509dc130a3daad
I20260926 22:52:41.977239 522 ts_tablet_manager.cc:1939] T a921ca9b8cac4d9caa509dc130a3daad P a8aed9c97fb64af79886c639aac5cb6b: Deleting consensus metadata
I20260926 22:52:41.977597 32488 catalog_manager.cc:4841] TS a8aed9c97fb64af79886c639aac5cb6b (127.13.148.3:35183): tablet a921ca9b8cac4d9caa509dc130a3daad (table test-table [id=357c81b3b7f34d7d96148464d9bb1ac3]) successfully deleted
I20260926 22:52:41.979681 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 355
I20260926 22:52:41.984097 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 32470
2026-09-26T22:52:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (787 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:42 dist-test-slave-98hb krb5kdc[526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:42 dist-test-slave-98hb krb5kdc[526](info): set up 2 sockets
Sep 26 22:52:42 dist-test-slave-98hb krb5kdc[526](info): commencing operation
krb5kdc: starting...
W20260926 22:52:42.933691 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:52:43.042747 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42547
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:42547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:43.120265 539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:43.120483 539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:43.120513 539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:43.121692 539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:43.121757 539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:43.121779 539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:43.121796 539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:43.121821 539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:43.123322 539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:42547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:42547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:43.123564 539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:43.123771 539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:43.128286 547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:43.128310 545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:43.128317 544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:43.128516 539 server_base.cc:1034] running on GCE node
I20260926 22:52:43.128768 539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:43.129067 539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:43.130234 539 hybrid_clock.cc:648] HybridClock initialized: now 1790463163130212 us; error 33 us; skew 500 ppm
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:43.132653 539 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:52:43.133131 539 webserver.cc:460] Webserver started at http://127.13.148.62:35225/ using document root <none> and password file <none>
I20260926 22:52:43.133323 539 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:43.133370 539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:43.133492 539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:43.134590 539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "b2acc7e4421b4c5e96c94d0cad45dd00"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "23f369e0646238e2c1324b0f90f0000c"
server_key_iv: "d3f97d267e2aaed91657dfbc5417f1b0"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.134927 539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "b2acc7e4421b4c5e96c94d0cad45dd00"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "23f369e0646238e2c1324b0f90f0000c"
server_key_iv: "d3f97d267e2aaed91657dfbc5417f1b0"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.136186 539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:52:43.136909 554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.137070 539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:43.137151 539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "b2acc7e4421b4c5e96c94d0cad45dd00"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "23f369e0646238e2c1324b0f90f0000c"
server_key_iv: "d3f97d267e2aaed91657dfbc5417f1b0"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.137214 539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:43.148331 539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:43.150204 539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:43.150337 539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:43.153846 539 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:42547
I20260926 22:52:43.153877 606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:42547 every 8 connection(s)
I20260926 22:52:43.154196 539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:43.154978 607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:43.156682 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 539
I20260926 22:52:43.156788 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:43.156963 13904 external_mini_cluster.cc:1428] Setting key 09d943ca4e4812c8eb186125bada2a26
I20260926 22:52:43.157246 607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00: Bootstrap starting.
I20260926 22:52:43.158159 607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:43.158458 607 log.cc:824] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:43.159143 607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00: No bootstrap required, opened a new log
I20260926 22:52:43.161296 607 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2acc7e4421b4c5e96c94d0cad45dd00" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42547 } }
I20260926 22:52:43.161458 607 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:43.161489 607 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2acc7e4421b4c5e96c94d0cad45dd00, State: Initialized, Role: FOLLOWER
I20260926 22:52:43.161609 607 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2acc7e4421b4c5e96c94d0cad45dd00" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42547 } }
I20260926 22:52:43.161689 607 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:43.161720 607 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:43.161824 607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:43.162462 607 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2acc7e4421b4c5e96c94d0cad45dd00" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42547 } }
I20260926 22:52:43.162550 607 leader_election.cc:302] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2acc7e4421b4c5e96c94d0cad45dd00; no voters:
I20260926 22:52:43.162701 607 leader_election.cc:288] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:43.162760 612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:43.162936 607 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:43.162962 612 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [term 1 LEADER]: Becoming Leader. State: Replica: b2acc7e4421b4c5e96c94d0cad45dd00, State: Running, Role: LEADER
I20260926 22:52:43.163043 612 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2acc7e4421b4c5e96c94d0cad45dd00" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42547 } }
I20260926 22:52:43.163371 614 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2acc7e4421b4c5e96c94d0cad45dd00. Latest consensus state: current_term: 1 leader_uuid: "b2acc7e4421b4c5e96c94d0cad45dd00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2acc7e4421b4c5e96c94d0cad45dd00" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42547 } } }
I20260926 22:52:43.163516 614 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:43.163759 621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:43.164242 621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:43.164412 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2acc7e4421b4c5e96c94d0cad45dd00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2acc7e4421b4c5e96c94d0cad45dd00" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 42547 } } }
I20260926 22:52:43.164688 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:43.166399 621 catalog_manager.cc:1347] Generated new cluster ID: 0263fd8712084313b9d9e1500ca14e53
I20260926 22:52:43.166440 621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:43.167837 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.157419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.13.148.62:42547)
0926 22:52:43.157572 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.157581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.157612 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:52:43.158172 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.158183 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.158245 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.158401 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:43.158959 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.160275 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.161016 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.161612 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.161718 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.164501 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:43.164505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:43.164505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:43.164517 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:43.166302 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.166706 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.166709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.166709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.166807 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.166997 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.166997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.166998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.167267 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:43.167387 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.167509 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.167631 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260926 22:52:43.176350 621 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:43.176837 621 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:43.179705 621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00: Generated new TSK 0
I20260926 22:52:43.179840 621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.148.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
Entry for principal kudu/127.13.148.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab.
I20260926 22:52:43.204805 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42547
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:43.281947 633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:43.282135 633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:43.282163 633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:43.283257 633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:43.283296 633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:43.283330 633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:52:43.284762 633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.1.keytab
--principal=kudu/127.13.148.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:43.284981 633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:43.285177 633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:43.289698 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:43.289793 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:43.289746 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:43.289908 633 server_base.cc:1034] running on GCE node
I20260926 22:52:43.290071 633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:43.290282 633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:43.291419 633 hybrid_clock.cc:648] HybridClock initialized: now 1790463163291411 us; error 27 us; skew 500 ppm
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:43.293666 633 init.cc:377] Logged in from keytab as kudu/127.13.148.1@KRBTEST.COM (short username kudu)
I20260926 22:52:43.294100 633 webserver.cc:460] Webserver started at http://127.13.148.1:37507/ using document root <none> and password file <none>
I20260926 22:52:43.294307 633 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:43.294354 633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:43.294477 633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:43.295476 633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "c8208d4e3b204dd3bafa155ab2a56158"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "7ec00213109411994a3f77763ccc0496"
server_key_iv: "e224a4613068915b22042f12fc99094a"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.295807 633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "c8208d4e3b204dd3bafa155ab2a56158"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "7ec00213109411994a3f77763ccc0496"
server_key_iv: "e224a4613068915b22042f12fc99094a"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.297043 633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:43.297762 648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.297909 633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:43.297981 633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "c8208d4e3b204dd3bafa155ab2a56158"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "7ec00213109411994a3f77763ccc0496"
server_key_iv: "e224a4613068915b22042f12fc99094a"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.298048 633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:43.309769 633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:43.311360 633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:43.311484 633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:43.311688 633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:43.311960 633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:43.311992 633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.312018 633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:43.312036 633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.317436 633 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:45235
I20260926 22:52:43.317477 761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:45235 every 8 connection(s)
I20260926 22:52:43.317792 633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:52:43.319170 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 633
I20260926 22:52:43.319289 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:52:43.319506 13904 external_mini_cluster.cc:1428] Setting key 54ea28393abe3bb360155d5c16e62ebc
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.1@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:43.327083 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.318463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:38883 (local address 127.13.148.62:42547)
0926 22:52:43.318529 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.318531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.318679 (+ 148us) server_negotiation.cc:407] Connection header received
0926 22:52:43.319237 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.319239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.319250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.319313 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:43.320417 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.320928 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.322119 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.322414 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.322480 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.324585 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:43.324586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:43.324586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:43.324593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:43.325801 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.326216 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.326216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.326217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.326260 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.326504 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.326504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.326504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.326608 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:43.326651 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.326955 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.327016 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.13.148.2@KRBTEST.COM; defaulting to no policy
I20260926 22:52:43.327316 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.318428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42547 (local address 127.13.148.1:38883)
0926 22:52:43.318571 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.318605 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:43.319142 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:43.319395 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:43.319400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:43.320056 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:43.320317 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.320324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.321011 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.321012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.322045 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.322048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.322500 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.322500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.322604 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.323033 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:43.323066 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:43.324482 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:43.325869 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:43.325877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:43.325879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:43.326151 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:43.326313 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:43.326314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:43.326314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:43.326445 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:43.326667 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:43.326673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:43.326832 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 22:52:43.326946 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 22:52:43.327096 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260926 22:52:43.327834 762 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42547
I20260926 22:52:43.327921 762 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:43.328078 762 heartbeater.cc:511] Master 127.13.148.62:42547 requested a full tablet report, sending...
I20260926 22:52:43.328583 571 ts_manager.cc:194] Registered new tserver with Master: c8208d4e3b204dd3bafa155ab2a56158 (127.13.148.1:45235)
I20260926 22:52:43.329176 571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.1@KRBTEST.COM'} at 127.13.148.1:38883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
Entry for principal kudu/127.13.148.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab.
I20260926 22:52:43.344092 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42547
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:43.422942 767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:43.423132 767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:43.423152 767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:43.424227 767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:43.424269 767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:43.424295 767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:52:43.425683 767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.2.keytab
--principal=kudu/127.13.148.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:43.425915 767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:43.426110 767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:43.430629 773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:43.430698 775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:43.430750 767 server_base.cc:1034] running on GCE node
W20260926 22:52:43.430640 772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:43.430943 767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:43.431169 767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:43.432333 767 hybrid_clock.cc:648] HybridClock initialized: now 1790463163432322 us; error 26 us; skew 500 ppm
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:43.439503 767 init.cc:377] Logged in from keytab as kudu/127.13.148.2@KRBTEST.COM (short username kudu)
I20260926 22:52:43.439939 767 webserver.cc:460] Webserver started at http://127.13.148.2:44563/ using document root <none> and password file <none>
I20260926 22:52:43.440169 767 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:43.440217 767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:43.440339 767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:43.441310 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "c3dcbe3fe24b48f7983db668465df194"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "9404ab99638a90e10c2892361dc84aa8"
server_key_iv: "2c4531d8e93cda46e06fd126dc162a95"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.441619 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "c3dcbe3fe24b48f7983db668465df194"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "9404ab99638a90e10c2892361dc84aa8"
server_key_iv: "2c4531d8e93cda46e06fd126dc162a95"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.442813 767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 22:52:43.443485 782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.443629 767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:43.443707 767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "c3dcbe3fe24b48f7983db668465df194"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "9404ab99638a90e10c2892361dc84aa8"
server_key_iv: "2c4531d8e93cda46e06fd126dc162a95"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.443766 767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:43.464752 767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:43.466707 767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:43.466840 767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:43.467049 767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:43.467360 767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:43.467392 767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.467420 767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:43.467442 767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.472692 767 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:36057
I20260926 22:52:43.472743 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:36057 every 8 connection(s)
I20260926 22:52:43.473026 767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.2@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:43.478453 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 767
I20260926 22:52:43.478555 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:52:43.478729 13904 external_mini_cluster.cc:1428] Setting key be2e81b349a0bacb2602b81c37e26082
I20260926 22:52:43.481467 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.473652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:39995 (local address 127.13.148.62:42547)
0926 22:52:43.473748 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.473750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.473919 (+ 169us) server_negotiation.cc:407] Connection header received
0926 22:52:43.474375 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.474376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.474385 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.474430 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:43.475274 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.475759 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.476718 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.476904 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.476963 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.478913 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:43.478914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:43.478915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:43.478922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:43.480129 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.480542 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.480542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.480543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.480614 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.480866 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.480867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.480867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.480990 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:43.481050 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.481291 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.481379 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 22:52:43.481537 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.473633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42547 (local address 127.13.148.2:39995)
0926 22:52:43.473807 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.473843 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:43.474302 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:43.474490 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:43.474492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:43.475018 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:43.475212 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.475216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.475833 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.475833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.476638 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.476640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.476970 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.476971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.477061 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.477453 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:43.477460 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:43.478832 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:43.480199 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:43.480206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:43.480208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:43.480477 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:43.480677 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:43.480677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:43.480678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:43.480811 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:43.481058 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:43.481063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:43.481178 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 22:52:43.481259 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 22:52:43.481373 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260926 22:52:43.481956 896 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42547
I20260926 22:52:43.482059 896 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:43.482232 896 heartbeater.cc:511] Master 127.13.148.62:42547 requested a full tablet report, sending...
I20260926 22:52:43.482563 571 ts_manager.cc:194] Registered new tserver with Master: c3dcbe3fe24b48f7983db668465df194 (127.13.148.2:36057)
I20260926 22:52:43.482911 571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.2@KRBTEST.COM'} at 127.13.148.2:39995
WARNING: no policy specified for kudu/127.13.148.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
Entry for principal kudu/127.13.148.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab.
I20260926 22:52:43.502497 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42547
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:43.588368 901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:43.588632 901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:43.588665 901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:43.589848 901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:43.589885 901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:43.589910 901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:52:43.591323 901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.3.keytab
--principal=kudu/127.13.148.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:42547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:43.591554 901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:43.591770 901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:43.597144 907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:43.597141 909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:43.597244 901 server_base.cc:1034] running on GCE node
W20260926 22:52:43.597375 906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:43.597563 901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:43.597800 901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:43.598932 901 hybrid_clock.cc:648] HybridClock initialized: now 1790463163598920 us; error 30 us; skew 500 ppm
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:43.601189 901 init.cc:377] Logged in from keytab as kudu/127.13.148.3@KRBTEST.COM (short username kudu)
I20260926 22:52:43.601604 901 webserver.cc:460] Webserver started at http://127.13.148.3:36317/ using document root <none> and password file <none>
I20260926 22:52:43.601820 901 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:43.601863 901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:43.601958 901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:43.602921 901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "ecefa7f4f7e740fdabb179d4d64ca398"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "17f5e0f0eb3dd94331c345df6b7e9ffc"
server_key_iv: "864da0c6d8815ad8f52e5ccc67d2871f"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.603201 901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "ecefa7f4f7e740fdabb179d4d64ca398"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "17f5e0f0eb3dd94331c345df6b7e9ffc"
server_key_iv: "864da0c6d8815ad8f52e5ccc67d2871f"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.604367 901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:52:43.604969 916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.605098 901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:43.605175 901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "ecefa7f4f7e740fdabb179d4d64ca398"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "17f5e0f0eb3dd94331c345df6b7e9ffc"
server_key_iv: "864da0c6d8815ad8f52e5ccc67d2871f"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.605247 901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:43.617180 901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:43.618961 901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:43.619091 901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:43.619289 901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:43.619570 901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:43.619601 901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.619628 901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:43.619652 901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.625098 901 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:42037
I20260926 22:52:43.625160 1029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:42037 every 8 connection(s)
I20260926 22:52:43.625432 901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:52:43.626737 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 901
I20260926 22:52:43.626859 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:52:43.627091 13904 external_mini_cluster.cc:1428] Setting key 3ddfcadac117f3691be96ff54154b5d6
Sep 26 22:52:43 dist-test-slave-98hb krb5kdc[526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463163, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.3@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:52:43.633888 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.626092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:43413 (local address 127.13.148.62:42547)
0926 22:52:43.626170 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.626172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.626303 (+ 131us) server_negotiation.cc:407] Connection header received
0926 22:52:43.626767 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.626768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.626780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.626845 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:43.628065 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.628590 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.629540 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.629763 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.629819 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.631689 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:43.631690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:43.631690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:43.631697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:43.632840 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.633181 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.633182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.633182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.633223 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.633395 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.633395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.633396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.633484 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:43.633526 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.633758 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.633823 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 22:52:43.634120 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.626062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:42547 (local address 127.13.148.3:43413)
0926 22:52:43.626194 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.626229 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:43.626680 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:43.626937 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:43.626942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:43.627680 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:43.627960 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.627967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.628674 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.628675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.629477 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.629479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.629824 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.629825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.629935 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.630355 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:52:43.630362 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:43.631612 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:43.632903 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:43.632908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:43.632910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:43.633123 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:43.633266 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:52:43.633266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:52:43.633267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:52:43.633355 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:52:43.633525 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:43.633528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:43.633633 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 22:52:43.633715 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 22:52:43.633876 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260926 22:52:43.634528 1030 heartbeater.cc:347] Connected to a master server at 127.13.148.62:42547
I20260926 22:52:43.634613 1030 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:43.634783 1030 heartbeater.cc:511] Master 127.13.148.62:42547 requested a full tablet report, sending...
I20260926 22:52:43.635115 571 ts_manager.cc:194] Registered new tserver with Master: ecefa7f4f7e740fdabb179d4d64ca398 (127.13.148.3:42037)
I20260926 22:52:43.635457 571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.148.3@KRBTEST.COM'} at 127.13.148.3:43413
I20260926 22:52:43.638680 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:52:43.645288 610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.640023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.13.148.62:42547)
0926 22:52:43.640081 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.640083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.640093 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:52:43.640169 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.640170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.640179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.640221 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:43.640735 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.641247 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.641988 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.642178 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.642227 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.643245 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:43.643247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:43.643247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:43.643253 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:43.644285 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.644579 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.644581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.644581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.644630 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:43.644797 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:43.644798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:43.644798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:43.644937 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:43.645022 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.645129 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.645203 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 22:52:43.646765 571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:52:43.648416 571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:52:43.654403 1038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.650321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45235 (local address 127.0.0.1:54468)
0926 22:52:43.650641 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.650648 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:43.650677 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:43.651195 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:43.651196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:43.651198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:43.651251 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.651252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.652404 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.652404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.653248 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.653250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.654109 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.654110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.654186 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.654200 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 22:52:43.654273 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 22:52:43.654322 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":62,"threads_started":1}
I20260926 22:52:43.654421 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.650559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.13.148.1:45235)
0926 22:52:43.651047 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.651051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.651060 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:52:43.651100 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.651102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.651117 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.651188 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:43.651317 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.652328 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.653324 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.654021 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.654102 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.654163 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.654296 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.654343 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":85,"threads_started":1}
I20260926 22:52:43.654814 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.650321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:36057 (local address 127.0.0.1:35572)
0926 22:52:43.650915 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.650924 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:43.650980 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:43.651400 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:43.651401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:43.651402 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:43.651435 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.651436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.652225 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.652225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.653489 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.653490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.654609 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.654609 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.654671 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.654674 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:43.654715 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 22:52:43.654751 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":7,"thread_start_us":55,"threads_started":1}
I20260926 22:52:43.654985 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.650561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60250 (local address 127.13.148.3:42037)
0926 22:52:43.650748 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.650752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.650762 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:52:43.650793 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.650794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.650807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.650872 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:43.651051 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.652159 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.653574 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.654267 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.654329 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.654381 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.654864 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.654914 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260926 22:52:43.655145 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.650755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.13.148.2:36057)
0926 22:52:43.650905 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.650910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.650967 (+ 57us) server_negotiation.cc:407] Connection header received
0926 22:52:43.651311 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.651314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.651330 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.651399 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:43.651477 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.652140 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.653796 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.654454 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.654518 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.654575 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.654970 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.655015 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260926 22:52:43.655187 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.650321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42037 (local address 127.0.0.1:60250)
0926 22:52:43.650485 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.650528 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:43.650578 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:43.650883 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:43.650887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:43.650889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:43.650950 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.650952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.652267 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.652268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.653488 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.653490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.654415 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.654416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.654778 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.654782 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:43.654837 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 22:52:43.655028 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":67,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260926 22:52:43.655175 696 tablet_service.cc:1467] Processing CreateTablet for tablet 485e362692894a7a85e7bf84f84fd7e1 (DEFAULT_TABLE table=test-table [id=c452190cf6ea4c3aad7252b163616c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:43.655527 696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485e362692894a7a85e7bf84f84fd7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:43.655735 830 tablet_service.cc:1467] Processing CreateTablet for tablet 485e362692894a7a85e7bf84f84fd7e1 (DEFAULT_TABLE table=test-table [id=c452190cf6ea4c3aad7252b163616c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:43.655836 964 tablet_service.cc:1467] Processing CreateTablet for tablet 485e362692894a7a85e7bf84f84fd7e1 (DEFAULT_TABLE table=test-table [id=c452190cf6ea4c3aad7252b163616c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:43.656044 830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485e362692894a7a85e7bf84f84fd7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:43.656098 964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485e362692894a7a85e7bf84f84fd7e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:43.658625 1044 tablet_bootstrap.cc:492] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: Bootstrap starting.
I20260926 22:52:43.658983 1046 tablet_bootstrap.cc:492] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Bootstrap starting.
I20260926 22:52:43.659260 1044 tablet_bootstrap.cc:654] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:43.659277 1045 tablet_bootstrap.cc:492] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158: Bootstrap starting.
I20260926 22:52:43.659662 1046 tablet_bootstrap.cc:654] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:43.659749 1044 log.cc:824] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:43.659855 1045 tablet_bootstrap.cc:654] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:43.660259 1046 log.cc:824] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:43.660389 1045 log.cc:824] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:43.661136 1044 tablet_bootstrap.cc:492] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: No bootstrap required, opened a new log
I20260926 22:52:43.661137 1045 tablet_bootstrap.cc:492] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158: No bootstrap required, opened a new log
I20260926 22:52:43.661208 1045 ts_tablet_manager.cc:1397] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:43.661208 1044 ts_tablet_manager.cc:1397] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:52:43.661216 1046 tablet_bootstrap.cc:492] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: No bootstrap required, opened a new log
I20260926 22:52:43.661283 1046 ts_tablet_manager.cc:1397] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:52:43.662691 1046 raft_consensus.cc:348] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.662825 1046 raft_consensus.cc:374] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:43.662858 1046 raft_consensus.cc:729] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3dcbe3fe24b48f7983db668465df194, State: Initialized, Role: FOLLOWER
I20260926 22:52:43.662985 1046 consensus_queue.cc:260] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.663223 1046 ts_tablet_manager.cc:1428] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 22:52:43.663128 1045 raft_consensus.cc:348] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.663128 1044 raft_consensus.cc:348] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.663288 1045 raft_consensus.cc:374] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:43.663335 1044 raft_consensus.cc:374] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:43.663348 896 heartbeater.cc:503] Master 127.13.148.62:42547 was elected leader, sending a full tablet report...
I20260926 22:52:43.663365 1045 raft_consensus.cc:729] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8208d4e3b204dd3bafa155ab2a56158, State: Initialized, Role: FOLLOWER
I20260926 22:52:43.663371 1044 raft_consensus.cc:729] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecefa7f4f7e740fdabb179d4d64ca398, State: Initialized, Role: FOLLOWER
I20260926 22:52:43.663476 1044 consensus_queue.cc:260] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.663590 1045 consensus_queue.cc:260] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.663738 1044 ts_tablet_manager.cc:1428] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 22:52:43.663877 1030 heartbeater.cc:503] Master 127.13.148.62:42547 was elected leader, sending a full tablet report...
I20260926 22:52:43.664192 762 heartbeater.cc:503] Master 127.13.148.62:42547 was elected leader, sending a full tablet report...
I20260926 22:52:43.664198 1045 ts_tablet_manager.cc:1428] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 22:52:43.723573 897 tablet.cc:2369] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:52:43.723896 1050 raft_consensus.cc:482] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:43.723974 1050 raft_consensus.cc:504] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.724352 1050 leader_election.cc:288] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecefa7f4f7e740fdabb179d4d64ca398 (127.13.148.3:42037), c8208d4e3b204dd3bafa155ab2a56158 (127.13.148.1:45235)
I20260926 22:52:43.727046 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.724678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:37743 (local address 127.13.148.3:42037)
0926 22:52:43.724719 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.724721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.724728 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:52:43.724751 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.724752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.724762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.724815 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:43.724959 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.725437 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.726365 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.726736 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.726798 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.726807 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.726940 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.726975 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:52:43.727064 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.724508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:42037 (local address 127.13.148.2:37743)
0926 22:52:43.724573 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.724584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:43.724629 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:43.724821 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:43.724822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:43.724823 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:43.724896 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.724897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.725524 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.725525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.726278 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.726280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.726807 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.726809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.726876 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.726879 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:43.726925 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:52:43.726967 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 22:52:43.727092 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.724693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:33263 (local address 127.13.148.1:45235)
0926 22:52:43.724737 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.724739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.724746 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:52:43.724797 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.724798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.724809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.724856 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:43.724995 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.725531 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.726429 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.726797 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.726857 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.726864 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.726983 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.727028 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:52:43.727293 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.724510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45235 (local address 127.13.148.2:33263)
0926 22:52:43.724683 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.724692 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:43.724739 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:43.724865 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:43.724866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:43.724867 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:43.724932 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.724933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.725601 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.725602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.726346 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:43.726348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.726863 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.726863 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:43.726927 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.726929 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 22:52:43.726973 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 22:52:43.727008 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260926 22:52:43.727397 984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e362692894a7a85e7bf84f84fd7e1" candidate_uuid: "c3dcbe3fe24b48f7983db668465df194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" is_pre_election: true
I20260926 22:52:43.727551 984 raft_consensus.cc:2454] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dcbe3fe24b48f7983db668465df194 in term 0.
I20260926 22:52:43.727592 716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e362692894a7a85e7bf84f84fd7e1" candidate_uuid: "c3dcbe3fe24b48f7983db668465df194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8208d4e3b204dd3bafa155ab2a56158" is_pre_election: true
I20260926 22:52:43.727744 716 raft_consensus.cc:2454] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3dcbe3fe24b48f7983db668465df194 in term 0.
I20260926 22:52:43.727751 783 leader_election.cc:302] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3dcbe3fe24b48f7983db668465df194, ecefa7f4f7e740fdabb179d4d64ca398; no voters:
I20260926 22:52:43.727861 1050 raft_consensus.cc:2781] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:52:43.727927 1050 raft_consensus.cc:482] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:52:43.727949 1050 raft_consensus.cc:3037] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:43.728631 1050 raft_consensus.cc:504] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.728734 1050 leader_election.cc:288] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [CANDIDATE]: Term 1 election: Requested vote from peers ecefa7f4f7e740fdabb179d4d64ca398 (127.13.148.3:42037), c8208d4e3b204dd3bafa155ab2a56158 (127.13.148.1:45235)
I20260926 22:52:43.728892 984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e362692894a7a85e7bf84f84fd7e1" candidate_uuid: "c3dcbe3fe24b48f7983db668465df194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecefa7f4f7e740fdabb179d4d64ca398"
I20260926 22:52:43.728906 716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485e362692894a7a85e7bf84f84fd7e1" candidate_uuid: "c3dcbe3fe24b48f7983db668465df194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8208d4e3b204dd3bafa155ab2a56158"
I20260926 22:52:43.728958 984 raft_consensus.cc:3037] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:43.728964 716 raft_consensus.cc:3037] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:43.729647 716 raft_consensus.cc:2454] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dcbe3fe24b48f7983db668465df194 in term 1.
I20260926 22:52:43.729658 984 raft_consensus.cc:2454] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3dcbe3fe24b48f7983db668465df194 in term 1.
I20260926 22:52:43.729817 783 leader_election.cc:302] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3dcbe3fe24b48f7983db668465df194, ecefa7f4f7e740fdabb179d4d64ca398; no voters:
I20260926 22:52:43.729899 1050 raft_consensus.cc:2781] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:43.730036 1050 raft_consensus.cc:686] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 1 LEADER]: Becoming Leader. State: Replica: c3dcbe3fe24b48f7983db668465df194, State: Running, Role: LEADER
I20260926 22:52:43.730125 1050 consensus_queue.cc:237] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } }
I20260926 22:52:43.730912 570 catalog_manager.cc:5495] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3dcbe3fe24b48f7983db668465df194 (127.13.148.2). New cstate: current_term: 1 leader_uuid: "c3dcbe3fe24b48f7983db668465df194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3dcbe3fe24b48f7983db668465df194" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 36057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 } health_report { overall_health: UNKNOWN } } }
I20260926 22:52:43.747323 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.745028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.13.148.2:36057)
0926 22:52:43.745082 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.745084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.745094 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:52:43.745129 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.745130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.745151 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.745210 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:52:43.745313 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.745823 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.746601 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.746805 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.746856 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.746999 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:52:43.747110 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:52:43.747176 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.747213 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.747231 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:52:43.748976 984 raft_consensus.cc:1261] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 1 FOLLOWER]: Refusing update from remote peer c3dcbe3fe24b48f7983db668465df194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:43.749010 716 raft_consensus.cc:1261] T 485e362692894a7a85e7bf84f84fd7e1 P c8208d4e3b204dd3bafa155ab2a56158 [term 1 FOLLOWER]: Refusing update from remote peer c3dcbe3fe24b48f7983db668465df194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:43.749346 1050 consensus_queue.cc:1035] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8208d4e3b204dd3bafa155ab2a56158" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 45235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:43.749480 1054 consensus_queue.cc:1035] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecefa7f4f7e740fdabb179d4d64ca398" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 42037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:43.751699 1059 mvcc.cc:205] Tried to move back new op lower bound from 7333737118713331712 to 7333737118639149056. Current Snapshot: MvccSnapshot[applied={T|T < 7333737118713331712}]
I20260926 22:52:43.752666 1058 mvcc.cc:205] Tried to move back new op lower bound from 7333737118713331712 to 7333737118639149056. Current Snapshot: MvccSnapshot[applied={T|T < 7333737118713331712}]
I20260926 22:52:43.753850 571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:43.753927 571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:43.754580 571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2acc7e4421b4c5e96c94d0cad45dd00: Sending DeleteTablet for 3 replicas of tablet 485e362692894a7a85e7bf84f84fd7e1
I20260926 22:52:43.754885 964 tablet_service.cc:1514] Processing DeleteTablet for tablet 485e362692894a7a85e7bf84f84fd7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:43 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:60250
I20260926 22:52:43.754907 830 tablet_service.cc:1514] Processing DeleteTablet for tablet 485e362692894a7a85e7bf84f84fd7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:43 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:35572
I20260926 22:52:43.755084 1066 tablet_replica.cc:330] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: stopping tablet replica
I20260926 22:52:43.755122 1067 tablet_replica.cc:330] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: stopping tablet replica
I20260926 22:52:43.755142 696 tablet_service.cc:1514] Processing DeleteTablet for tablet 485e362692894a7a85e7bf84f84fd7e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:43 UTC) from {username='kudu', principal='kudu/127.13.148.62@KRBTEST.COM'} at 127.0.0.1:54468
I20260926 22:52:43.755185 1066 raft_consensus.cc:2229] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:43.755257 1067 raft_consensus.cc:2229] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:52:43.755278 1066 raft_consensus.cc:2258] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:43.755370 1067 raft_consensus.cc:2258] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:43.755702 1066 ts_tablet_manager.cc:1905] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:43.755885 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 633
I20260926 22:52:43.756942 1067 ts_tablet_manager.cc:1905] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:43.757297 1066 ts_tablet_manager.cc:1918] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:43.757368 1066 log.cc:1196] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/485e362692894a7a85e7bf84f84fd7e1
I20260926 22:52:43.757598 1066 ts_tablet_manager.cc:1939] T 485e362692894a7a85e7bf84f84fd7e1 P ecefa7f4f7e740fdabb179d4d64ca398: Deleting consensus metadata
I20260926 22:52:43.758023 555 catalog_manager.cc:4841] TS ecefa7f4f7e740fdabb179d4d64ca398 (127.13.148.3:42037): tablet 485e362692894a7a85e7bf84f84fd7e1 (table test-table [id=c452190cf6ea4c3aad7252b163616c2d]) successfully deleted
I20260926 22:52:43.758483 1067 ts_tablet_manager.cc:1918] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:43.758527 1067 log.cc:1196] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/485e362692894a7a85e7bf84f84fd7e1
I20260926 22:52:43.758694 1067 ts_tablet_manager.cc:1939] T 485e362692894a7a85e7bf84f84fd7e1 P c3dcbe3fe24b48f7983db668465df194: Deleting consensus metadata
I20260926 22:52:43.759234 556 catalog_manager.cc:4841] TS c3dcbe3fe24b48f7983db668465df194 (127.13.148.2:36057): tablet 485e362692894a7a85e7bf84f84fd7e1 (table test-table [id=c452190cf6ea4c3aad7252b163616c2d]) successfully deleted
W20260926 22:52:43.760547 558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:45235 (error 108)
I20260926 22:52:43.760790 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 767
I20260926 22:52:43.761366 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.760898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:45235 (local address 127.0.0.1:54476)
0926 22:52:43.761299 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:43.761335 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
W20260926 22:52:43.761519 558 catalog_manager.cc:4584] TS c8208d4e3b204dd3bafa155ab2a56158 (127.13.148.1:45235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 485e362692894a7a85e7bf84f84fd7e1: Network error: Client connection negotiation failed: client connection to 127.13.148.1:45235: connect: Connection refused (error 111)
I20260926 22:52:43.765781 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 901
I20260926 22:52:43.770418 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 539
2026-09-26T22:52:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1787 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T22:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:43Z Disabled control of system clock
I20260926 22:52:43.795692 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39105
--webserver_interface=127.13.148.62
--webserver_port=0
--builtin_ntp_servers=127.13.148.20:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:43.873581 1073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:43.873818 1073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:43.873842 1073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:43.874980 1073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:43.875027 1073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:43.875038 1073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:43.875049 1073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:43.875059 1073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:43.876376 1073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:43.876605 1073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:43.876866 1073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:43.881665 1081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:43.881675 1078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:43.881712 1073 server_base.cc:1034] running on GCE node
W20260926 22:52:43.881682 1079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:43.882114 1073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:43.882370 1073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:43.883498 1073 hybrid_clock.cc:648] HybridClock initialized: now 1790463163883490 us; error 26 us; skew 500 ppm
I20260926 22:52:43.887746 1073 webserver.cc:460] Webserver started at http://127.13.148.62:46353/ using document root <none> and password file <none>
I20260926 22:52:43.887953 1073 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:43.887995 1073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:43.888118 1073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:43.889154 1073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "cf5af0b3e890d1032769cf39ef3ab8e8"
server_key_iv: "865dff16768da3a025380f4b8928fba3"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.889469 1073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "cf5af0b3e890d1032769cf39ef3ab8e8"
server_key_iv: "865dff16768da3a025380f4b8928fba3"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.890666 1073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:52:43.891294 1087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.891420 1073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:43.891497 1073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28"
format_stamp: "Formatted at 2026-09-26 22:52:43 on dist-test-slave-98hb"
server_key: "cf5af0b3e890d1032769cf39ef3ab8e8"
server_key_iv: "865dff16768da3a025380f4b8928fba3"
server_key_version: "encryptionkey@0"
I20260926 22:52:43.891562 1073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:43.907199 1073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:43.907455 1073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:43.907581 1073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:43.911115 1073 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39105
I20260926 22:52:43.911151 1139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39105 every 8 connection(s)
I20260926 22:52:43.911442 1073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:43.912330 1140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:43.914539 1140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28: Bootstrap starting.
I20260926 22:52:43.915069 1140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:43.915324 1140 log.cc:824] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:43.915910 1140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28: No bootstrap required, opened a new log
I20260926 22:52:43.917455 1140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39105 } }
I20260926 22:52:43.917587 1140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:43.917609 1140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c9ad29c063441fb8e5a4cfca6e9aa28, State: Initialized, Role: FOLLOWER
I20260926 22:52:43.917770 1140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39105 } }
I20260926 22:52:43.917838 1140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:43.917860 1140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:43.917904 1140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:43.918375 1140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39105 } }
I20260926 22:52:43.918462 1140 leader_election.cc:302] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c9ad29c063441fb8e5a4cfca6e9aa28; no voters:
I20260926 22:52:43.918627 1140 leader_election.cc:288] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:43.918668 1143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:43.918831 1143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [term 1 LEADER]: Becoming Leader. State: Replica: 9c9ad29c063441fb8e5a4cfca6e9aa28, State: Running, Role: LEADER
I20260926 22:52:43.918849 1140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:43.918927 1143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39105 } }
I20260926 22:52:43.919276 1145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c9ad29c063441fb8e5a4cfca6e9aa28. Latest consensus state: current_term: 1 leader_uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39105 } } }
I20260926 22:52:43.919344 1144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c9ad29c063441fb8e5a4cfca6e9aa28" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39105 } } }
I20260926 22:52:43.919359 1145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:43.919404 1144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:43.919524 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 1073
I20260926 22:52:43.919637 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:43.919636 1150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:43.919838 13904 external_mini_cluster.cc:1428] Setting key e570da99c2bafb290d43e513c51092c2
I20260926 22:52:43.920089 1150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:43.921793 1150 catalog_manager.cc:1347] Generated new cluster ID: 575a3839ad9d49fc80318f360013fac9
I20260926 22:52:43.921845 1150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:43.926084 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:43.920608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.13.148.62:39105)
0926 22:52:43.920900 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:43.920909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:43.921104 (+ 195us) server_negotiation.cc:407] Connection header received
0926 22:52:43.921218 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:43.921225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:43.921306 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:43.921450 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:43.921645 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.922987 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.924090 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:43.924682 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:43.924779 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:43.925423 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:43.925426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:43.925427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:43.925606 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:43.925645 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:43.925658 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:43.925713 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:43.925812 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 22:52:43.925882 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260926 22:52:43.934376 1150 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:43.934819 1150 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:43.943813 1150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28: Generated new TSK 0
I20260926 22:52:43.943988 1150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:43.950592 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.1:0
--local_ip_for_outbound_sockets=127.13.148.1
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39105
--builtin_ntp_servers=127.13.148.20:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:44.028874 1164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:44.029071 1164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:44.029093 1164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:44.030277 1164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:44.030316 1164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:44.030340 1164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.1
I20260926 22:52:44.031636 1164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.148.1
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.1
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:44.031847 1164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:44.032110 1164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:44.037595 1172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:44.037606 1169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:44.037611 1170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:44.037912 1164 server_base.cc:1034] running on GCE node
I20260926 22:52:44.038101 1164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:44.038319 1164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:44.039467 1164 hybrid_clock.cc:648] HybridClock initialized: now 1790463164039454 us; error 29 us; skew 500 ppm
I20260926 22:52:44.040644 1164 webserver.cc:460] Webserver started at http://127.13.148.1:42781/ using document root <none> and password file <none>
I20260926 22:52:44.040861 1164 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:44.040912 1164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:44.041039 1164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:44.042115 1164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/data/instance:
uuid: "f2febb8f1f054a43bd916cd155dcaec7"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "92e988e39cb57607184720206af57c23"
server_key_iv: "e1e777e223df60af89b55b6a7c4f5044"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.042429 1164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance:
uuid: "f2febb8f1f054a43bd916cd155dcaec7"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "92e988e39cb57607184720206af57c23"
server_key_iv: "e1e777e223df60af89b55b6a7c4f5044"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.043629 1164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:44.044371 1178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.044564 1164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:44.044649 1164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/wal
uuid: "f2febb8f1f054a43bd916cd155dcaec7"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "92e988e39cb57607184720206af57c23"
server_key_iv: "e1e777e223df60af89b55b6a7c4f5044"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.044708 1164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:44.063567 1164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:44.063817 1164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:44.063911 1164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:44.064087 1164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:44.064366 1164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:44.064404 1164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.064426 1164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:44.064440 1164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.069562 1164 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.1:34023
I20260926 22:52:44.069631 1291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.1:34023 every 8 connection(s)
I20260926 22:52:44.069927 1164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/data/info.pb
I20260926 22:52:44.074573 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.070592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.1:32949 (local address 127.13.148.62:39105)
0926 22:52:44.070635 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.070637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.070852 (+ 215us) server_negotiation.cc:407] Connection header received
0926 22:52:44.071301 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.071302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.071311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.071371 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:44.071987 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.072496 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.073397 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.073593 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.073645 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.074390 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:44.074390 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:44.074391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:44.074396 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:44.074406 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:44.074410 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:44.074445 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.074492 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.074513 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:52:44.074762 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.070547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39105 (local address 127.13.148.1:32949)
0926 22:52:44.070732 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.070770 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:44.071232 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:44.071426 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:44.071429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:44.071437 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:44.071915 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.071920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.072573 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.072574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.073323 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.073325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.073655 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.073656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.073785 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.074280 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:44.074281 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:44.074308 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:44.074331 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:44.074446 (+ 115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:44.074448 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:44.074451 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:44.074494 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 22:52:44.074577 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260926 22:52:44.075130 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 1164
I20260926 22:52:44.075250 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-0/wal/instance
I20260926 22:52:44.075271 1292 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39105
I20260926 22:52:44.075361 1292 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:44.075457 13904 external_mini_cluster.cc:1428] Setting key b8c3a2c9b69f5c2d326d0a0a40df5609
I20260926 22:52:44.075539 1292 heartbeater.cc:511] Master 127.13.148.62:39105 requested a full tablet report, sending...
I20260926 22:52:44.076289 1104 ts_manager.cc:194] Registered new tserver with Master: f2febb8f1f054a43bd916cd155dcaec7 (127.13.148.1:34023)
I20260926 22:52:44.076316 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.2:0
--local_ip_for_outbound_sockets=127.13.148.2
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39105
--builtin_ntp_servers=127.13.148.20:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 22:52:44.076947 1104 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.1:32949
W20260926 22:52:44.155000 1295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:44.155192 1295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:44.155211 1295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:44.156350 1295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:44.156388 1295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:44.156411 1295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.2
I20260926 22:52:44.157720 1295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.148.2
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.2
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:44.157956 1295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:44.158226 1295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:44.162845 1303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:44.162858 1300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:44.162858 1301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:44.162901 1295 server_base.cc:1034] running on GCE node
I20260926 22:52:44.163261 1295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:44.163493 1295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:44.164624 1295 hybrid_clock.cc:648] HybridClock initialized: now 1790463164164616 us; error 29 us; skew 500 ppm
I20260926 22:52:44.165719 1295 webserver.cc:460] Webserver started at http://127.13.148.2:45267/ using document root <none> and password file <none>
I20260926 22:52:44.165977 1295 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:44.166024 1295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:44.166142 1295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:44.167115 1295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/data/instance:
uuid: "d78fb7521978462a849bc674c3c03885"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "853bf28c4b670559078a568c58fa9684"
server_key_iv: "91e5778693767233426a5437917e5d1e"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.167434 1295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance:
uuid: "d78fb7521978462a849bc674c3c03885"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "853bf28c4b670559078a568c58fa9684"
server_key_iv: "91e5778693767233426a5437917e5d1e"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.168586 1295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:52:44.169269 1309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.169416 1295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:44.169494 1295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/wal
uuid: "d78fb7521978462a849bc674c3c03885"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "853bf28c4b670559078a568c58fa9684"
server_key_iv: "91e5778693767233426a5437917e5d1e"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.169559 1295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:44.189136 1295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:44.189416 1295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:44.189546 1295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:44.189761 1295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:44.190027 1295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:44.190056 1295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.190079 1295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:44.190090 1295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.195277 1295 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.2:39445
I20260926 22:52:44.195346 1422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.2:39445 every 8 connection(s)
I20260926 22:52:44.195634 1295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/data/info.pb
I20260926 22:52:44.200438 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.196237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.2:50587 (local address 127.13.148.62:39105)
0926 22:52:44.196282 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.196284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.196532 (+ 248us) server_negotiation.cc:407] Connection header received
0926 22:52:44.197032 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.197034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.197046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.197094 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:44.197757 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.198244 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.199191 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.199376 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.199431 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.200183 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:44.200184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:44.200184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:44.200189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:44.200203 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:44.200208 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:44.200248 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.200324 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.200356 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:52:44.200634 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.196236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39105 (local address 127.13.148.2:50587)
0926 22:52:44.196405 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.196448 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:44.196935 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:44.197166 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:44.197170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:44.197181 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:44.197652 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.197657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.198329 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.198330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.199109 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.199112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.199448 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.199449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.199578 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.200031 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:44.200031 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:44.200058 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:44.200105 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:44.200255 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:44.200260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:44.200264 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:44.200323 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 22:52:44.200423 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260926 22:52:44.201083 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 1295
I20260926 22:52:44.201108 1423 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39105
I20260926 22:52:44.201187 1423 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:44.201191 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/wal/instance
I20260926 22:52:44.201361 1423 heartbeater.cc:511] Master 127.13.148.62:39105 requested a full tablet report, sending...
I20260926 22:52:44.201367 13904 external_mini_cluster.cc:1428] Setting key af11d8a6614d2f732da07ca672d0bcae
I20260926 22:52:44.201771 1103 ts_manager.cc:194] Registered new tserver with Master: d78fb7521978462a849bc674c3c03885 (127.13.148.2:39445)
I20260926 22:52:44.202184 1103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.2:50587
I20260926 22:52:44.202250 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.148.3:0
--local_ip_for_outbound_sockets=127.13.148.3
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39105
--builtin_ntp_servers=127.13.148.20:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 22:52:44.279454 1426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:44.279639 1426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:44.279659 1426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:44.280746 1426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:44.280783 1426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:44.280810 1426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.148.3
I20260926 22:52:44.282152 1426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.148.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.148.3
--webserver_port=0
--tserver_master_addrs=127.13.148.62:39105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.148.3
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:44.282368 1426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:44.282624 1426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:44.287103 1431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:44.287112 1432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:44.287197 1434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:44.287217 1426 server_base.cc:1034] running on GCE node
I20260926 22:52:44.287519 1426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:44.287750 1426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:44.288890 1426 hybrid_clock.cc:648] HybridClock initialized: now 1790463164288877 us; error 29 us; skew 500 ppm
I20260926 22:52:44.290040 1426 webserver.cc:460] Webserver started at http://127.13.148.3:38045/ using document root <none> and password file <none>
I20260926 22:52:44.290256 1426 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:44.290303 1426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:44.290421 1426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:44.291419 1426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/data/instance:
uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "a6b0cdd223d9338f429488811d206449"
server_key_iv: "9c41d36df2834687e5783afda300f775"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.291738 1426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance:
uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "a6b0cdd223d9338f429488811d206449"
server_key_iv: "9c41d36df2834687e5783afda300f775"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.292866 1426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:44.293481 1440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.293623 1426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:44.293699 1426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/wal
uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa"
format_stamp: "Formatted at 2026-09-26 22:52:44 on dist-test-slave-98hb"
server_key: "a6b0cdd223d9338f429488811d206449"
server_key_iv: "9c41d36df2834687e5783afda300f775"
server_key_version: "encryptionkey@0"
I20260926 22:52:44.293784 1426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:44.314769 1426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:44.315047 1426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:44.315176 1426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:44.315368 1426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 22:52:44.315696 1426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 22:52:44.315729 1426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.315757 1426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 22:52:44.315779 1426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:44.321269 1426 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.3:44075
I20260926 22:52:44.321308 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.3:44075 every 8 connection(s)
I20260926 22:52:44.321630 1426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/data/info.pb
I20260926 22:52:44.326373 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.322322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:36271 (local address 127.13.148.62:39105)
0926 22:52:44.322369 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.322371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.322612 (+ 241us) server_negotiation.cc:407] Connection header received
0926 22:52:44.323086 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.323087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.323098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.323157 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:44.323774 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.324256 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.325139 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.325321 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.325369 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.326142 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:44.326143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:44.326143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:44.326147 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:44.326167 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:44.326171 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:44.326211 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.326275 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.326310 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:52:44.326454 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 1426
I20260926 22:52:44.326560 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/wal/instance
I20260926 22:52:44.326627 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.322319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:39105 (local address 127.13.148.3:36271)
0926 22:52:44.322490 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.322522 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:44.322981 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:44.323221 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:44.323224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:44.323232 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:52:44.323712 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.323716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.324314 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.324315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.325076 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.325078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.325379 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.325380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.325480 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.326009 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 22:52:44.326009 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:52:44.326036 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 22:52:44.326059 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:52:44.326220 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:52:44.326222 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:52:44.326225 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:44.326276 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:52:44.326396 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260926 22:52:44.326779 13904 external_mini_cluster.cc:1428] Setting key 8c9ae7f809f319a568bea2ab370a4e63
I20260926 22:52:44.327041 1554 heartbeater.cc:347] Connected to a master server at 127.13.148.62:39105
I20260926 22:52:44.327131 1554 heartbeater.cc:464] Registering TS with master...
I20260926 22:52:44.327316 1554 heartbeater.cc:511] Master 127.13.148.62:39105 requested a full tablet report, sending...
I20260926 22:52:44.327636 1103 ts_manager.cc:194] Registered new tserver with Master: 9d47ee658d2a4ca5b65dd048ffcd34fa (127.13.148.3:44075)
I20260926 22:52:44.327991 1103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.148.3:36271
I20260926 22:52:44.338361 13904 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 22:52:44.342311 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.339632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.13.148.62:39105)
0926 22:52:44.339702 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.339704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.339748 (+ 44us) server_negotiation.cc:407] Connection header received
0926 22:52:44.339812 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.339813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.339824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.339868 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:44.340006 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.340485 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.341218 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.341397 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.341448 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.342082 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:44.342083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:44.342083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:44.342088 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:44.342101 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:44.342105 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:44.342146 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.342204 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.342227 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 22:52:44.343747 1103 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 22:52:44.345225 1103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 22:52:44.351089 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.347082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.3:44075 (local address 127.0.0.1:41842)
0926 22:52:44.347391 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.347415 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:44.347481 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:44.347767 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:44.347768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:44.347770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:44.347811 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.347813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.348788 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.348789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.349981 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.349984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.350815 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.350816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.350889 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.350904 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 22:52:44.350966 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:52:44.351017 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":67,"threads_started":1}
I20260926 22:52:44.351104 1565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.347241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41842 (local address 127.13.148.3:44075)
0926 22:52:44.347423 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.347427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.347575 (+ 148us) server_negotiation.cc:407] Connection header received
0926 22:52:44.347618 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.347621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.347636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.347699 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:44.347880 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.348720 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.350068 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.350734 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.350808 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.350907 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.350979 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.351015 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260926 22:52:44.351279 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.347264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34023 (local address 127.0.0.1:39774)
0926 22:52:44.347479 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.347488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:44.347539 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:44.347830 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:44.347831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:44.347833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:44.347864 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.347865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.349120 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.349121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.350151 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.350153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.351060 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.351061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.351119 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.351123 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 22:52:44.351174 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:52:44.351213 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":55,"threads_started":1}
I20260926 22:52:44.351414 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.347374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.13.148.1:34023)
0926 22:52:44.347611 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.347615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.347625 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:52:44.347661 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.347663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.347677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.347746 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:44.347934 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.349046 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.350219 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.350894 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.350951 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.351027 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.351328 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.351358 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260926 22:52:44.352051 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.347072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39445 (local address 127.0.0.1:33536)
0926 22:52:44.347374 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.347415 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:44.347481 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:44.347725 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:44.347727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:44.347729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:44.347789 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.347791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.350213 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.350213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.350874 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.350875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.351834 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.351834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.351896 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.351899 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:44.351945 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:52:44.351980 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":64,"threads_started":1}
I20260926 22:52:44.352078 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.347160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.13.148.2:39445)
0926 22:52:44.347310 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.347315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.347582 (+ 267us) server_negotiation.cc:407] Connection header received
0926 22:52:44.347623 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.347625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.347639 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.347709 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:44.347872 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.349027 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.350958 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.351665 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.351722 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.351803 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.351969 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.352005 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260926 22:52:44.352116 1226 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6feafe346949388ee89877341fe584 (DEFAULT_TABLE table=test-table [id=6061342f940340c6989cb6f4fa263047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:44.352115 1488 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6feafe346949388ee89877341fe584 (DEFAULT_TABLE table=test-table [id=6061342f940340c6989cb6f4fa263047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:44.352453 1226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6feafe346949388ee89877341fe584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:44.352453 1488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6feafe346949388ee89877341fe584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:44.355674 1568 tablet_bootstrap.cc:492] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7: Bootstrap starting.
I20260926 22:52:44.355952 1569 tablet_bootstrap.cc:492] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: Bootstrap starting.
I20260926 22:52:44.356263 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6feafe346949388ee89877341fe584 (DEFAULT_TABLE table=test-table [id=6061342f940340c6989cb6f4fa263047]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 22:52:44.356389 1568 tablet_bootstrap.cc:654] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:44.356505 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6feafe346949388ee89877341fe584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:44.356612 1569 tablet_bootstrap.cc:654] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:44.356686 1568 log.cc:824] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:44.356884 1569 log.cc:824] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:44.357445 1568 tablet_bootstrap.cc:492] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7: No bootstrap required, opened a new log
I20260926 22:52:44.357503 1568 ts_tablet_manager.cc:1397] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:44.358968 1572 tablet_bootstrap.cc:492] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: Bootstrap starting.
I20260926 22:52:44.359112 1569 tablet_bootstrap.cc:492] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: No bootstrap required, opened a new log
I20260926 22:52:44.359180 1569 ts_tablet_manager.cc:1397] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 22:52:44.359583 1572 tablet_bootstrap.cc:654] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:44.359639 1568 raft_consensus.cc:348] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.359812 1568 raft_consensus.cc:374] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:44.359854 1568 raft_consensus.cc:729] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2febb8f1f054a43bd916cd155dcaec7, State: Initialized, Role: FOLLOWER
I20260926 22:52:44.359858 1572 log.cc:824] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:44.359969 1568 consensus_queue.cc:260] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.360229 1568 ts_tablet_manager.cc:1428] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:52:44.360262 1292 heartbeater.cc:503] Master 127.13.148.62:39105 was elected leader, sending a full tablet report...
I20260926 22:52:44.360486 1572 tablet_bootstrap.cc:492] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: No bootstrap required, opened a new log
I20260926 22:52:44.360549 1572 ts_tablet_manager.cc:1397] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 22:52:44.361445 1569 raft_consensus.cc:348] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.361557 1569 raft_consensus.cc:374] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:44.361577 1569 raft_consensus.cc:729] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d47ee658d2a4ca5b65dd048ffcd34fa, State: Initialized, Role: FOLLOWER
I20260926 22:52:44.361654 1569 consensus_queue.cc:260] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.361903 1569 ts_tablet_manager.cc:1428] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 22:52:44.362017 1554 heartbeater.cc:503] Master 127.13.148.62:39105 was elected leader, sending a full tablet report...
I20260926 22:52:44.362636 1572 raft_consensus.cc:348] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.362782 1572 raft_consensus.cc:374] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:44.362809 1572 raft_consensus.cc:729] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78fb7521978462a849bc674c3c03885, State: Initialized, Role: FOLLOWER
I20260926 22:52:44.362895 1572 consensus_queue.cc:260] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.363106 1572 ts_tablet_manager.cc:1428] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 22:52:44.363169 1423 heartbeater.cc:503] Master 127.13.148.62:39105 was elected leader, sending a full tablet report...
I20260926 22:52:44.384948 1575 raft_consensus.cc:482] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 22:52:44.385043 1575 raft_consensus.cc:504] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.385315 1575 leader_election.cc:288] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d78fb7521978462a849bc674c3c03885 (127.13.148.2:39445), f2febb8f1f054a43bd916cd155dcaec7 (127.13.148.1:34023)
I20260926 22:52:44.388932 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.385530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:50489 (local address 127.13.148.2:39445)
0926 22:52:44.385573 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.385575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.385584 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:52:44.385666 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.385667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.385676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.385720 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:44.385918 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.386654 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.388057 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.388605 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.388668 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.388675 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.388828 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.388865 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:52:44.388932 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.385632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.148.3:46219 (local address 127.13.148.1:34023)
0926 22:52:44.385666 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.385668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.385785 (+ 117us) server_negotiation.cc:407] Connection header received
0926 22:52:44.385834 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.385835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.385843 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.385884 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 22:52:44.385998 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.386670 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.388057 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.388614 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.388669 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.388675 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.388835 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.388868 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:52:44.388957 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.385466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39445 (local address 127.13.148.3:50489)
0926 22:52:44.385521 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.385548 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:44.385586 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:44.385787 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:44.385788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:44.385789 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:44.385852 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.385854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.386727 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.386728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.387969 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.387972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.388681 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.388682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.388765 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.388768 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:44.388817 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:52:44.388859 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":32}
I20260926 22:52:44.388968 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.385556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34023 (local address 127.13.148.3:46219)
0926 22:52:44.385707 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.385715 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:52:44.385789 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:52:44.385886 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:52:44.385887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:52:44.385888 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 22:52:44.385942 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.385943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.386736 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.386737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.387972 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:52:44.387973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.388675 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.388676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:52:44.388769 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.388772 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 22:52:44.388826 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 22:52:44.388856 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":31,"thread_start_us":98,"threads_started":1}
I20260926 22:52:44.389325 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6feafe346949388ee89877341fe584" candidate_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78fb7521978462a849bc674c3c03885" is_pre_election: true
I20260926 22:52:44.389330 1246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6feafe346949388ee89877341fe584" candidate_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2febb8f1f054a43bd916cd155dcaec7" is_pre_election: true
I20260926 22:52:44.389443 1246 raft_consensus.cc:2454] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d47ee658d2a4ca5b65dd048ffcd34fa in term 0.
I20260926 22:52:44.389449 1377 raft_consensus.cc:2454] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d47ee658d2a4ca5b65dd048ffcd34fa in term 0.
I20260926 22:52:44.389657 1444 leader_election.cc:302] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d47ee658d2a4ca5b65dd048ffcd34fa, f2febb8f1f054a43bd916cd155dcaec7; no voters:
I20260926 22:52:44.389798 1575 raft_consensus.cc:2781] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 22:52:44.389865 1575 raft_consensus.cc:482] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 22:52:44.389889 1575 raft_consensus.cc:3037] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:44.390563 1575 raft_consensus.cc:504] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.390667 1575 leader_election.cc:288] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [CANDIDATE]: Term 1 election: Requested vote from peers d78fb7521978462a849bc674c3c03885 (127.13.148.2:39445), f2febb8f1f054a43bd916cd155dcaec7 (127.13.148.1:34023)
I20260926 22:52:44.390859 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6feafe346949388ee89877341fe584" candidate_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d78fb7521978462a849bc674c3c03885"
I20260926 22:52:44.390878 1246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6feafe346949388ee89877341fe584" candidate_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2febb8f1f054a43bd916cd155dcaec7"
I20260926 22:52:44.390944 1377 raft_consensus.cc:3037] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:44.390944 1246 raft_consensus.cc:3037] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:44.391812 1246 raft_consensus.cc:2454] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d47ee658d2a4ca5b65dd048ffcd34fa in term 1.
I20260926 22:52:44.391812 1377 raft_consensus.cc:2454] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d47ee658d2a4ca5b65dd048ffcd34fa in term 1.
I20260926 22:52:44.391958 1444 leader_election.cc:302] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d47ee658d2a4ca5b65dd048ffcd34fa, f2febb8f1f054a43bd916cd155dcaec7; no voters:
I20260926 22:52:44.392042 1575 raft_consensus.cc:2781] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:44.392170 1575 raft_consensus.cc:686] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 1 LEADER]: Becoming Leader. State: Replica: 9d47ee658d2a4ca5b65dd048ffcd34fa, State: Running, Role: LEADER
I20260926 22:52:44.392244 1575 consensus_queue.cc:237] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } }
I20260926 22:52:44.392983 1103 catalog_manager.cc:5495] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d47ee658d2a4ca5b65dd048ffcd34fa (127.13.148.3). New cstate: current_term: 1 leader_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d47ee658d2a4ca5b65dd048ffcd34fa" member_type: VOTER last_known_addr { host: "127.13.148.3" port: 44075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 } health_report { overall_health: UNKNOWN } } }
I20260926 22:52:44.398877 1565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.396816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.13.148.3:44075)
0926 22:52:44.396885 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:44.396887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:44.396896 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:52:44.396923 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:44.396924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:44.396942 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:44.397045 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:52:44.397159 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.397594 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.398321 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:44.398495 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:44.398548 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:44.398648 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:52:44.398729 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:52:44.398781 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:44.398792 (+ 11us) server_negotiation.cc:299] Negotiation successful
0926 22:52:44.398805 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 22:52:44.400372 1377 raft_consensus.cc:1261] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 1 FOLLOWER]: Refusing update from remote peer 9d47ee658d2a4ca5b65dd048ffcd34fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:44.400372 1246 raft_consensus.cc:1261] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 1 FOLLOWER]: Refusing update from remote peer 9d47ee658d2a4ca5b65dd048ffcd34fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:52:44.400691 1578 consensus_queue.cc:1035] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2febb8f1f054a43bd916cd155dcaec7" member_type: VOTER last_known_addr { host: "127.13.148.1" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:44.400785 1575 consensus_queue.cc:1035] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d78fb7521978462a849bc674c3c03885" member_type: VOTER last_known_addr { host: "127.13.148.2" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:52:44.402769 1581 mvcc.cc:205] Tried to move back new op lower bound from 7333737121381781504 to 7333737121351233536. Current Snapshot: MvccSnapshot[applied={T|T < 7333737121381781504}]
I20260926 22:52:44.403599 1583 mvcc.cc:205] Tried to move back new op lower bound from 7333737121381781504 to 7333737121351233536. Current Snapshot: MvccSnapshot[applied={T|T < 7333737121381781504}]
I20260926 22:52:44.404824 1104 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:44.404910 1104 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 22:52:44.405561 1104 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c9ad29c063441fb8e5a4cfca6e9aa28: Sending DeleteTablet for 3 replicas of tablet 1d6feafe346949388ee89877341fe584
I20260926 22:52:44.405867 1357 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6feafe346949388ee89877341fe584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:44 UTC) from {username='slave'} at 127.0.0.1:33536
I20260926 22:52:44.405900 1226 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6feafe346949388ee89877341fe584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:44 UTC) from {username='slave'} at 127.0.0.1:39774
I20260926 22:52:44.406090 1592 tablet_replica.cc:330] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7: stopping tablet replica
I20260926 22:52:44.406212 1592 raft_consensus.cc:2229] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:44.406193 1488 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6feafe346949388ee89877341fe584 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 22:52:44 UTC) from {username='slave'} at 127.0.0.1:41842
I20260926 22:52:44.406317 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 1164
I20260926 22:52:44.406360 1592 raft_consensus.cc:2258] T 1d6feafe346949388ee89877341fe584 P f2febb8f1f054a43bd916cd155dcaec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:44.406379 1591 tablet_replica.cc:330] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: stopping tablet replica
I20260926 22:52:44.406461 1591 raft_consensus.cc:2229] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 22:52:44.406545 1591 raft_consensus.cc:2258] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:44.407222 1593 tablet_replica.cc:330] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: stopping tablet replica
I20260926 22:52:44.407303 1593 raft_consensus.cc:2229] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 1 LEADER]: Raft consensus shutting down.
I20260926 22:52:44.407399 1593 raft_consensus.cc:2258] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 22:52:44.407835 1591 ts_tablet_manager.cc:1905] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:44.409412 1591 ts_tablet_manager.cc:1918] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:44.409832 1593 ts_tablet_manager.cc:1905] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 22:52:44.411384 1593 ts_tablet_manager.cc:1918] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 22:52:44.411499 1591 log.cc:1196] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-1/wal/wals/1d6feafe346949388ee89877341fe584
W20260926 22:52:44.411720 1091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.148.1:34023 (error 108)
I20260926 22:52:44.411769 1591 ts_tablet_manager.cc:1939] T 1d6feafe346949388ee89877341fe584 P d78fb7521978462a849bc674c3c03885: Deleting consensus metadata
I20260926 22:52:44.411955 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 1295
I20260926 22:52:44.412226 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.412123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.1:34023 (local address 127.0.0.1:39778)
0926 22:52:44.412164 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.412196 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.1:34023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 22:52:44.412308 1091 catalog_manager.cc:4584] TS f2febb8f1f054a43bd916cd155dcaec7 (127.13.148.1:34023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d6feafe346949388ee89877341fe584: Network error: Client connection negotiation failed: client connection to 127.13.148.1:34023: connect: Connection refused (error 111)
I20260926 22:52:44.413609 1593 log.cc:1196] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: Deleting WAL directory at /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790462935413609-13904-0/minicluster-data/ts-2/wal/wals/1d6feafe346949388ee89877341fe584
I20260926 22:52:44.413923 1593 ts_tablet_manager.cc:1939] T 1d6feafe346949388ee89877341fe584 P 9d47ee658d2a4ca5b65dd048ffcd34fa: Deleting consensus metadata
I20260926 22:52:44.414330 1089 catalog_manager.cc:4841] TS 9d47ee658d2a4ca5b65dd048ffcd34fa (127.13.148.3:44075): tablet 1d6feafe346949388ee89877341fe584 (table test-table [id=6061342f940340c6989cb6f4fa263047]) successfully deleted
I20260926 22:52:44.416437 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 1426
I20260926 22:52:44.417076 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:44.416529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.2:39445 (local address 127.0.0.1:33542)
0926 22:52:44.417015 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 22:52:44.417047 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.2:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465}
I20260926 22:52:44.420521 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 1073
2026-09-26T22:52:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (649 ms)
[----------] 6 tests from EncryptionPolicyTest (7589 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:44 dist-test-slave-98hb krb5kdc[1597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:44 dist-test-slave-98hb krb5kdc[1597](info): set up 2 sockets
Sep 26 22:52:44 dist-test-slave-98hb krb5kdc[1597](info): commencing operation
krb5kdc: starting...
W20260926 22:52:45.365847 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:45 dist-test-slave-98hb krb5kdc[1597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:45.476042 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:45.553676 1610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:45.553887 1610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:45.553907 1610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:45.554976 1610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:45.555008 1610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:45.555022 1610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:45.555032 1610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:45.555047 1610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:45.556452 1610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:45.556633 1610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:45.556795 1610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:45.561192 1618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:45.561216 1610 server_base.cc:1034] running on GCE node
W20260926 22:52:45.561281 1616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:45.561223 1615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:45.561584 1610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:45.561867 1610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:45.563038 1610 hybrid_clock.cc:648] HybridClock initialized: now 1790463165563015 us; error 31 us; skew 500 ppm
Sep 26 22:52:45 dist-test-slave-98hb krb5kdc[1597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:45.565366 1610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:45.565797 1610 webserver.cc:460] Webserver started at http://127.0.0.1:36117/ using document root <none> and password file <none>
I20260926 22:52:45.565995 1610 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:45.566051 1610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:45.566169 1610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:45.567153 1610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "47191c67a9e84e53af2127090c2c9046"
format_stamp: "Formatted at 2026-09-26 22:52:45 on dist-test-slave-98hb"
server_key: "6c30aace4fd8fe8eaeb7f0d1c144580d"
server_key_iv: "ce2d017f6d3da9bc85184fc6d0274726"
server_key_version: "encryptionkey@0"
I20260926 22:52:45.567483 1610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "47191c67a9e84e53af2127090c2c9046"
format_stamp: "Formatted at 2026-09-26 22:52:45 on dist-test-slave-98hb"
server_key: "6c30aace4fd8fe8eaeb7f0d1c144580d"
server_key_iv: "ce2d017f6d3da9bc85184fc6d0274726"
server_key_version: "encryptionkey@0"
I20260926 22:52:45.568768 1610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 22:52:45.569412 1625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:45.569576 1610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:45.569654 1610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "47191c67a9e84e53af2127090c2c9046"
format_stamp: "Formatted at 2026-09-26 22:52:45 on dist-test-slave-98hb"
server_key: "6c30aace4fd8fe8eaeb7f0d1c144580d"
server_key_iv: "ce2d017f6d3da9bc85184fc6d0274726"
server_key_version: "encryptionkey@0"
I20260926 22:52:45.569713 1610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:45.576766 1610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:45.578442 1610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:45.578567 1610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:45.582109 1610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45417
I20260926 22:52:45.582139 1677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45417 every 8 connection(s)
I20260926 22:52:45.582432 1610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:45.583211 1678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:45.585299 1678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046: Bootstrap starting.
I20260926 22:52:45.585856 1678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:45.586105 1678 log.cc:824] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:45.586711 1678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046: No bootstrap required, opened a new log
I20260926 22:52:45.588212 1678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47191c67a9e84e53af2127090c2c9046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260926 22:52:45.588343 1678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:45.588376 1678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47191c67a9e84e53af2127090c2c9046, State: Initialized, Role: FOLLOWER
I20260926 22:52:45.588514 1678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47191c67a9e84e53af2127090c2c9046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260926 22:52:45.588595 1678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:45.588630 1678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:45.588675 1678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:45.589154 1678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47191c67a9e84e53af2127090c2c9046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260926 22:52:45.589241 1678 leader_election.cc:302] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47191c67a9e84e53af2127090c2c9046; no voters:
I20260926 22:52:45.589386 1678 leader_election.cc:288] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:45.589414 1681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:45.589534 1681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [term 1 LEADER]: Becoming Leader. State: Replica: 47191c67a9e84e53af2127090c2c9046, State: Running, Role: LEADER
I20260926 22:52:45.589624 1678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:45.589653 1681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47191c67a9e84e53af2127090c2c9046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } }
I20260926 22:52:45.589939 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 1610
I20260926 22:52:45.590063 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:45.590150 1683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47191c67a9e84e53af2127090c2c9046. Latest consensus state: current_term: 1 leader_uuid: "47191c67a9e84e53af2127090c2c9046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47191c67a9e84e53af2127090c2c9046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } } }
I20260926 22:52:45.590229 1683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:45.590265 13904 external_mini_cluster.cc:1428] Setting key 461a80e465f2d4a4849ddafbeb6e7227
I20260926 22:52:45.590519 1690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:45.591014 1690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:45.590376 1682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47191c67a9e84e53af2127090c2c9046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47191c67a9e84e53af2127090c2c9046" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45417 } } }
I20260926 22:52:45.591338 1682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:45.592677 1690 catalog_manager.cc:1347] Generated new cluster ID: d0a68c30a5b84c3f9efe964c37ab9deb
I20260926 22:52:45.592726 1690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:52:45 dist-test-slave-98hb krb5kdc[1597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 22:52:45.599545 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:45.590912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52119 (local address 127.0.0.1:45417)
0926 22:52:45.591050 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:45.591060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:45.591162 (+ 102us) server_negotiation.cc:407] Connection header received
0926 22:52:45.591279 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:45.591287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:45.591445 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:45.591592 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:45.592296 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:45.593280 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:45.594060 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:45.594466 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:45.594556 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:45.596646 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:45.596649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:45.596649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:45.596658 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:45.598143 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:45.598489 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:45.598491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:45.598492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:45.598584 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:45.598780 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:45.598781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:45.598782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:45.598995 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:45.599154 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:45.599268 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 22:52:45.599367 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
I20260926 22:52:45.600510 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:45.607163 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:45.601725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35363 (local address 127.0.0.1:45417)
0926 22:52:45.601776 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:45.601778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:45.601878 (+ 100us) server_negotiation.cc:407] Connection header received
0926 22:52:45.601929 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:45.601929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:45.601938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:45.601979 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:45.602425 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:45.602881 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:45.603511 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:45.603730 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:45.603781 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:45.604787 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:45.604788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:45.604789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:45.604795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:45.606039 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:45.606462 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:45.606463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:45.606463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:45.606516 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:45.606719 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:45.606720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:45.606720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:45.606817 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:45.606898 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:45.607019 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 22:52:45.607085 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 22:52:45.613390 1690 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:45.614100 1690 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:45.621908 1690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47191c67a9e84e53af2127090c2c9046: Generated new TSK 0
I20260926 22:52:45.622097 1690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:45.627890 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 1610
W20260926 22:52:45.631711 1595 connection.cc:535] client connection to 127.0.0.1:45417 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45417): Connection reset by peer (error 104)
2026-09-26T22:52:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:45 dist-test-slave-98hb krb5kdc[1709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:45 dist-test-slave-98hb krb5kdc[1709](info): set up 2 sockets
Sep 26 22:52:45 dist-test-slave-98hb krb5kdc[1709](info): commencing operation
krb5kdc: starting...
W20260926 22:52:46.570003 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:46 dist-test-slave-98hb krb5kdc[1709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:46.678017 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:46.755110 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:46.755294 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:46.755314 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:46.756455 1722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:46.756497 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:46.756510 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:46.756520 1722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:46.756542 1722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:46.757974 1722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45081
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:46.758188 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:46.758386 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:46.762933 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:46.762929 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:46.763195 1722 server_base.cc:1034] running on GCE node
W20260926 22:52:46.762925 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:46.763456 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:46.763725 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:46.764859 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790463166764842 us; error 32 us; skew 500 ppm
Sep 26 22:52:46 dist-test-slave-98hb krb5kdc[1709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:46.767184 1722 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:46.767551 1722 webserver.cc:460] Webserver started at http://127.0.0.1:44855/ using document root <none> and password file <none>
I20260926 22:52:46.767735 1722 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:46.767773 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:46.767864 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:46.768823 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "730276a88d884634918875dfa9be654a"
format_stamp: "Formatted at 2026-09-26 22:52:46 on dist-test-slave-98hb"
server_key: "bc6cc41517a7545972bebe68caeef488"
server_key_iv: "31f4ad58b8508aa0ca1c98546d2d5b63"
server_key_version: "encryptionkey@0"
I20260926 22:52:46.769109 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "730276a88d884634918875dfa9be654a"
format_stamp: "Formatted at 2026-09-26 22:52:46 on dist-test-slave-98hb"
server_key: "bc6cc41517a7545972bebe68caeef488"
server_key_iv: "31f4ad58b8508aa0ca1c98546d2d5b63"
server_key_version: "encryptionkey@0"
I20260926 22:52:46.770287 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:46.770888 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:46.771021 1722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:46.771126 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "730276a88d884634918875dfa9be654a"
format_stamp: "Formatted at 2026-09-26 22:52:46 on dist-test-slave-98hb"
server_key: "bc6cc41517a7545972bebe68caeef488"
server_key_iv: "31f4ad58b8508aa0ca1c98546d2d5b63"
server_key_version: "encryptionkey@0"
I20260926 22:52:46.771183 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:46.787336 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:46.789369 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:46.789495 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:46.793134 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45081
I20260926 22:52:46.793174 1789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45081 every 8 connection(s)
I20260926 22:52:46.793474 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:46.794286 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:46.796391 1790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a: Bootstrap starting.
I20260926 22:52:46.796885 1790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:46.797096 1790 log.cc:824] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:46.797684 1790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a: No bootstrap required, opened a new log
I20260926 22:52:46.799168 1790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730276a88d884634918875dfa9be654a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45081 } }
I20260926 22:52:46.799299 1790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:46.799320 1790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730276a88d884634918875dfa9be654a, State: Initialized, Role: FOLLOWER
I20260926 22:52:46.799427 1790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730276a88d884634918875dfa9be654a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45081 } }
I20260926 22:52:46.799496 1790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:46.799525 1790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:46.799561 1790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:46.799984 1790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730276a88d884634918875dfa9be654a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45081 } }
I20260926 22:52:46.800064 1790 leader_election.cc:302] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 730276a88d884634918875dfa9be654a; no voters:
I20260926 22:52:46.800180 1790 leader_election.cc:288] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:46.800223 1793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:46.800377 1790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:46.800406 1793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [term 1 LEADER]: Becoming Leader. State: Replica: 730276a88d884634918875dfa9be654a, State: Running, Role: LEADER
I20260926 22:52:46.800524 1793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730276a88d884634918875dfa9be654a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45081 } }
I20260926 22:52:46.800881 1794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "730276a88d884634918875dfa9be654a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730276a88d884634918875dfa9be654a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45081 } } }
I20260926 22:52:46.800971 1794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:46.801059 1795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 730276a88d884634918875dfa9be654a. Latest consensus state: current_term: 1 leader_uuid: "730276a88d884634918875dfa9be654a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "730276a88d884634918875dfa9be654a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45081 } } }
I20260926 22:52:46.801165 1795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:46.801188 1802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:46.801782 1802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:46.801872 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 1722
I20260926 22:52:46.801986 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:46.802193 13904 external_mini_cluster.cc:1428] Setting key 9646ee3f3d8d7e7358949442e0c4dea2
I20260926 22:52:46.803363 1802 catalog_manager.cc:1347] Generated new cluster ID: 65ca21083ecc4b058113d3a859078456
I20260926 22:52:46.803416 1802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:46.810401 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:46.802586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39317 (local address 127.0.0.1:45081)
0926 22:52:46.803026 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:46.803035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:46.803065 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:52:46.803155 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:46.803160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:46.803200 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:46.803316 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:46.805890 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:46.807249 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:46.808374 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:46.808957 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:46.809051 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:46.809715 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:46.809719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:46.809720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:46.809948 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:46.809984 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:46.809997 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:46.810079 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:46.810163 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 22:52:46.810188 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":82,"threads_started":1}
I20260926 22:52:46.811208 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:46.815912 1812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:46.812537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45563 (local address 127.0.0.1:45081)
0926 22:52:46.812588 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:46.812590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:46.812711 (+ 121us) server_negotiation.cc:407] Connection header received
0926 22:52:46.812740 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:46.812741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:46.812760 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:46.812822 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:46.812939 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:46.813699 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:46.814696 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:46.815005 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:46.815071 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:46.815682 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:46.815683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:46.815684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:46.815689 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:46.815705 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:46.815709 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:46.815767 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:46.815823 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 22:52:46.815835 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:52:46.838151 1802 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:46.838657 1802 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:46.847012 1802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 730276a88d884634918875dfa9be654a: Generated new TSK 0
I20260926 22:52:46.847188 1802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:46.850638 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 1722
W20260926 22:52:46.854815 1707 connection.cc:535] client connection to 127.0.0.1:45081 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45081): Connection reset by peer (error 104)
2026-09-26T22:52:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:46 dist-test-slave-98hb krb5kdc[1821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:46 dist-test-slave-98hb krb5kdc[1821](info): set up 2 sockets
Sep 26 22:52:46 dist-test-slave-98hb krb5kdc[1821](info): commencing operation
krb5kdc: starting...
W20260926 22:52:47.797894 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:47 dist-test-slave-98hb krb5kdc[1821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:47.908317 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:47.985365 1834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:47.985560 1834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:47.985581 1834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:47.986670 1834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:47.986704 1834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:47.986716 1834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:47.986727 1834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:47.986737 1834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:47.988121 1834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:47.988298 1834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:47.988459 1834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:47.993016 1842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:47.993055 1840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:47.993139 1839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:47.993129 1834 server_base.cc:1034] running on GCE node
I20260926 22:52:47.993456 1834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:47.993717 1834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:47.994871 1834 hybrid_clock.cc:648] HybridClock initialized: now 1790463167994861 us; error 26 us; skew 500 ppm
Sep 26 22:52:47 dist-test-slave-98hb krb5kdc[1821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:47.997198 1834 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:47.997645 1834 webserver.cc:460] Webserver started at http://127.0.0.1:39083/ using document root <none> and password file <none>
I20260926 22:52:47.997889 1834 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:47.997937 1834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:47.998087 1834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:47.999107 1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "71668d0287f144a99cd20334b0c37575"
format_stamp: "Formatted at 2026-09-26 22:52:47 on dist-test-slave-98hb"
server_key: "3e189b0ca1cab7e01813b99c00fab52b"
server_key_iv: "996814a989231612be7ce60dfd993682"
server_key_version: "encryptionkey@0"
I20260926 22:52:47.999421 1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "71668d0287f144a99cd20334b0c37575"
format_stamp: "Formatted at 2026-09-26 22:52:47 on dist-test-slave-98hb"
server_key: "3e189b0ca1cab7e01813b99c00fab52b"
server_key_iv: "996814a989231612be7ce60dfd993682"
server_key_version: "encryptionkey@0"
I20260926 22:52:48.000669 1834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:52:48.001386 1849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:48.001549 1834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:48.001632 1834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "71668d0287f144a99cd20334b0c37575"
format_stamp: "Formatted at 2026-09-26 22:52:47 on dist-test-slave-98hb"
server_key: "3e189b0ca1cab7e01813b99c00fab52b"
server_key_iv: "996814a989231612be7ce60dfd993682"
server_key_version: "encryptionkey@0"
I20260926 22:52:48.001696 1834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:48.005303 1834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:48.007170 1834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:48.007294 1834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:48.010957 1834 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33115
I20260926 22:52:48.010987 1901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33115 every 8 connection(s)
I20260926 22:52:48.011276 1834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:48.012055 1902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:48.012374 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 1834
I20260926 22:52:48.012470 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:48.012663 13904 external_mini_cluster.cc:1428] Setting key 1432b1268be09dca323993b62ad09f01
I20260926 22:52:48.014732 1902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575: Bootstrap starting.
I20260926 22:52:48.014955 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:48.013150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34827 (local address 127.0.0.1:33115)
0926 22:52:48.013323 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:48.013331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:48.013357 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:52:48.013792 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:48.013800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:48.013828 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:48.013942 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:48.014547 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:48.014551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:48.014551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:48.014555 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:48.014597 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:48.014610 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:48.014667 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:48.014747 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 22:52:48.014781 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260926 22:52:48.015391 1902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:48.015678 1902 log.cc:824] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:48.015838 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:48.016330 1902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575: No bootstrap required, opened a new log
I20260926 22:52:48.018147 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:48.017147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43077 (local address 127.0.0.1:33115)
0926 22:52:48.017188 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:48.017189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:48.017292 (+ 103us) server_negotiation.cc:407] Connection header received
0926 22:52:48.017348 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:48.017348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:48.017353 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:48.017396 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:48.017966 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:48.017968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:48.017968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:48.017972 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:48.017984 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:48.017988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:48.018032 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:48.018078 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 22:52:48.018088 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:52:48.018328 1902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71668d0287f144a99cd20334b0c37575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260926 22:52:48.018466 1902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:48.018508 1902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71668d0287f144a99cd20334b0c37575, State: Initialized, Role: FOLLOWER
I20260926 22:52:48.018671 1902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71668d0287f144a99cd20334b0c37575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260926 22:52:48.018748 1902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:48.018774 1902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:48.018805 1902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:48.019277 1902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71668d0287f144a99cd20334b0c37575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260926 22:52:48.019362 1902 leader_election.cc:302] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71668d0287f144a99cd20334b0c37575; no voters:
I20260926 22:52:48.019479 1902 leader_election.cc:288] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:48.019518 1912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:48.019691 1902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:48.019696 1912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [term 1 LEADER]: Becoming Leader. State: Replica: 71668d0287f144a99cd20334b0c37575, State: Running, Role: LEADER
I20260926 22:52:48.019850 1912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71668d0287f144a99cd20334b0c37575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260926 22:52:48.020220 1913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71668d0287f144a99cd20334b0c37575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71668d0287f144a99cd20334b0c37575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } } }
I20260926 22:52:48.020229 1914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71668d0287f144a99cd20334b0c37575. Latest consensus state: current_term: 1 leader_uuid: "71668d0287f144a99cd20334b0c37575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71668d0287f144a99cd20334b0c37575" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } } }
I20260926 22:52:48.020344 1913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:48.020350 1914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:48.020592 1921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:48.021021 1921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:48.022419 1921 catalog_manager.cc:1347] Generated new cluster ID: 015c4bec2fb140feb88a6a75347c953a
I20260926 22:52:48.022459 1921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:48.028599 1921 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:48.028752 1921 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:48.039773 1921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71668d0287f144a99cd20334b0c37575: Generated new TSK 0
I20260926 22:52:48.039958 1921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:48.043905 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 1834
2026-09-26T22:52:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:48 dist-test-slave-98hb krb5kdc[1933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:48 dist-test-slave-98hb krb5kdc[1933](info): set up 2 sockets
Sep 26 22:52:48 dist-test-slave-98hb krb5kdc[1933](info): commencing operation
krb5kdc: starting...
W20260926 22:52:48.989815 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:49 dist-test-slave-98hb krb5kdc[1933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:49.097747 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:49.175421 1946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:49.175642 1946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:49.175673 1946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:49.176800 1946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:49.176841 1946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:49.176862 1946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:49.176877 1946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:49.176893 1946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:49.178369 1946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:49.178606 1946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:49.178812 1946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:49.183301 1951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:49.183320 1952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:49.183301 1954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:49.183382 1946 server_base.cc:1034] running on GCE node
I20260926 22:52:49.183750 1946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:49.184027 1946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:49.185166 1946 hybrid_clock.cc:648] HybridClock initialized: now 1790463169185155 us; error 29 us; skew 500 ppm
Sep 26 22:52:49 dist-test-slave-98hb krb5kdc[1933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:49.192487 1946 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:49.193049 1946 webserver.cc:460] Webserver started at http://127.0.0.1:36019/ using document root <none> and password file <none>
I20260926 22:52:49.193298 1946 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:49.193349 1946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:49.193478 1946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:49.194728 1946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "4d84bafd87ed4a6da22cb9ea541f8532"
format_stamp: "Formatted at 2026-09-26 22:52:49 on dist-test-slave-98hb"
server_key: "8af33249b75bc6fa9d2c471fce832517"
server_key_iv: "96ca64ab3e5d608d9948df6da00a34d3"
server_key_version: "encryptionkey@0"
I20260926 22:52:49.195057 1946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "4d84bafd87ed4a6da22cb9ea541f8532"
format_stamp: "Formatted at 2026-09-26 22:52:49 on dist-test-slave-98hb"
server_key: "8af33249b75bc6fa9d2c471fce832517"
server_key_iv: "96ca64ab3e5d608d9948df6da00a34d3"
server_key_version: "encryptionkey@0"
I20260926 22:52:49.196285 1946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:52:49.196888 1961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:49.197032 1946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:49.197118 1946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "4d84bafd87ed4a6da22cb9ea541f8532"
format_stamp: "Formatted at 2026-09-26 22:52:49 on dist-test-slave-98hb"
server_key: "8af33249b75bc6fa9d2c471fce832517"
server_key_iv: "96ca64ab3e5d608d9948df6da00a34d3"
server_key_version: "encryptionkey@0"
I20260926 22:52:49.197175 1946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:49.219712 1946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:49.221632 1946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:49.221792 1946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:49.225365 1946 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38685
I20260926 22:52:49.225409 2013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38685 every 8 connection(s)
I20260926 22:52:49.225720 1946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:49.226504 2014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:49.228618 2014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532: Bootstrap starting.
I20260926 22:52:49.229103 2014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:49.229322 2014 log.cc:824] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:49.229926 2014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532: No bootstrap required, opened a new log
I20260926 22:52:49.231463 2014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d84bafd87ed4a6da22cb9ea541f8532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260926 22:52:49.231622 2014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:49.231642 2014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d84bafd87ed4a6da22cb9ea541f8532, State: Initialized, Role: FOLLOWER
I20260926 22:52:49.231626 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 1946
I20260926 22:52:49.231727 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:49.231768 2014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d84bafd87ed4a6da22cb9ea541f8532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260926 22:52:49.231820 2014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:49.231854 2014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:49.231896 2014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:49.231904 13904 external_mini_cluster.cc:1428] Setting key a0d918639d71ecd0b7066d35e4a90f3d
I20260926 22:52:49.232486 2014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d84bafd87ed4a6da22cb9ea541f8532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260926 22:52:49.232577 2014 leader_election.cc:302] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d84bafd87ed4a6da22cb9ea541f8532; no voters:
I20260926 22:52:49.232736 2014 leader_election.cc:288] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:49.232769 2018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:49.232887 2018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [term 1 LEADER]: Becoming Leader. State: Replica: 4d84bafd87ed4a6da22cb9ea541f8532, State: Running, Role: LEADER
I20260926 22:52:49.232970 2018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d84bafd87ed4a6da22cb9ea541f8532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } }
I20260926 22:52:49.232976 2014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:49.234098 2020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d84bafd87ed4a6da22cb9ea541f8532. Latest consensus state: current_term: 1 leader_uuid: "4d84bafd87ed4a6da22cb9ea541f8532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d84bafd87ed4a6da22cb9ea541f8532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } } }
I20260926 22:52:49.234200 2020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:49.234297 2021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d84bafd87ed4a6da22cb9ea541f8532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d84bafd87ed4a6da22cb9ea541f8532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38685 } } }
I20260926 22:52:49.234375 2021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532 [sys.catalog]: This master's current role is: LEADER
W20260926 22:52:49.234683 2034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:52:49.234736 2034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:52:49.234829 2035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:49.234959 2035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:49.236496 2035 catalog_manager.cc:1347] Generated new cluster ID: afdb0662f4f245bf9278bd94e746c169
I20260926 22:52:49.236546 2035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:52:49 dist-test-slave-98hb krb5kdc[1933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 22:52:49.242296 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:49.232628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55573 (local address 127.0.0.1:38685)
0926 22:52:49.232851 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:49.232860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:49.232891 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:52:49.233091 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:49.233115 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:49.233201 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:49.233334 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:49.234071 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:49.235253 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:49.236278 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:49.236811 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:49.236905 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:49.239080 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:49.239084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:49.239084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:49.239095 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:49.240908 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:49.241230 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:49.241233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:49.241233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:49.241320 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:49.241488 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:49.241488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:49.241488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:49.241716 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:49.241884 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:49.241985 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 22:52:49.242115 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260926 22:52:49.243137 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:49.253387 2035 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:49.254134 2035 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:49.255578 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:49.246003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50467 (local address 127.0.0.1:38685)
0926 22:52:49.246034 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:49.246035 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:49.249868 (+ 3833us) server_negotiation.cc:407] Connection header received
0926 22:52:49.249928 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:49.249930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:49.249938 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:49.249982 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:49.250449 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:49.250923 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:49.251555 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:49.251751 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:49.251803 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:49.252791 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:49.252792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:49.252793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:49.252799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:49.254352 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:49.254744 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:49.254745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:49.254746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:49.254800 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:49.254973 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:49.254974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:49.254974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:49.255085 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:49.255169 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:49.255389 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 22:52:49.255484 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":15}
I20260926 22:52:49.261958 2035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d84bafd87ed4a6da22cb9ea541f8532: Generated new TSK 0
I20260926 22:52:49.262215 2035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:49.267381 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 1946
2026-09-26T22:52:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:49 dist-test-slave-98hb krb5kdc[2045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:49 dist-test-slave-98hb krb5kdc[2045](info): set up 2 sockets
Sep 26 22:52:49 dist-test-slave-98hb krb5kdc[2045](info): commencing operation
krb5kdc: starting...
W20260926 22:52:50.217912 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:50 dist-test-slave-98hb krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:50.323788 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:50.402323 2058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:50.402529 2058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:50.402551 2058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:50.403797 2058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:50.403841 2058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:50.403854 2058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:50.403865 2058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:50.403875 2058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:50.405306 2058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:50.405494 2058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:50.405666 2058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:50.410107 2064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:50.410087 2063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:50.410187 2066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:50.410225 2058 server_base.cc:1034] running on GCE node
I20260926 22:52:50.410518 2058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:50.410784 2058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:50.411914 2058 hybrid_clock.cc:648] HybridClock initialized: now 1790463170411902 us; error 24 us; skew 500 ppm
Sep 26 22:52:50 dist-test-slave-98hb krb5kdc[2045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:50.414163 2058 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:50.414546 2058 webserver.cc:460] Webserver started at http://127.0.0.1:44481/ using document root <none> and password file <none>
I20260926 22:52:50.414728 2058 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:50.414767 2058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:50.414852 2058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:50.415813 2058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "e8e05e97036343a8be26207ae1fd6034"
format_stamp: "Formatted at 2026-09-26 22:52:50 on dist-test-slave-98hb"
server_key: "dc5ea649da7ae3befc17b7ea85a67c7a"
server_key_iv: "11b759d39b2aa101821c34ac2cadc20b"
server_key_version: "encryptionkey@0"
I20260926 22:52:50.416114 2058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "e8e05e97036343a8be26207ae1fd6034"
format_stamp: "Formatted at 2026-09-26 22:52:50 on dist-test-slave-98hb"
server_key: "dc5ea649da7ae3befc17b7ea85a67c7a"
server_key_iv: "11b759d39b2aa101821c34ac2cadc20b"
server_key_version: "encryptionkey@0"
I20260926 22:52:50.417233 2058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:50.417897 2073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:50.418066 2058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:50.418159 2058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "e8e05e97036343a8be26207ae1fd6034"
format_stamp: "Formatted at 2026-09-26 22:52:50 on dist-test-slave-98hb"
server_key: "dc5ea649da7ae3befc17b7ea85a67c7a"
server_key_iv: "11b759d39b2aa101821c34ac2cadc20b"
server_key_version: "encryptionkey@0"
I20260926 22:52:50.418220 2058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:50.448895 2058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:50.450981 2058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:50.451123 2058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:50.454691 2058 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33693
I20260926 22:52:50.454734 2125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33693 every 8 connection(s)
I20260926 22:52:50.455049 2058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:50.455847 2126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:50.457901 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2058
I20260926 22:52:50.458029 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:50.458216 13904 external_mini_cluster.cc:1428] Setting key f6748c63f050c994d63d9dc0af8c5650
I20260926 22:52:50.458250 2126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034: Bootstrap starting.
I20260926 22:52:50.458890 2126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:50.459157 2126 log.cc:824] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:50.459898 2126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034: No bootstrap required, opened a new log
I20260926 22:52:50.461936 2126 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e05e97036343a8be26207ae1fd6034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33693 } }
I20260926 22:52:50.462069 2126 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:50.462090 2126 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e05e97036343a8be26207ae1fd6034, State: Initialized, Role: FOLLOWER
I20260926 22:52:50.462206 2126 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e05e97036343a8be26207ae1fd6034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33693 } }
I20260926 22:52:50.462266 2126 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:50.462289 2126 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:50.462333 2126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:50.462824 2126 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e05e97036343a8be26207ae1fd6034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33693 } }
I20260926 22:52:50.462916 2126 leader_election.cc:302] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8e05e97036343a8be26207ae1fd6034; no voters:
I20260926 22:52:50.463069 2126 leader_election.cc:288] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:50.463100 2131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:50.463258 2131 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [term 1 LEADER]: Becoming Leader. State: Replica: e8e05e97036343a8be26207ae1fd6034, State: Running, Role: LEADER
I20260926 22:52:50.463315 2126 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:50.463340 2131 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e05e97036343a8be26207ae1fd6034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33693 } }
I20260926 22:52:50.463685 2133 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e05e97036343a8be26207ae1fd6034. Latest consensus state: current_term: 1 leader_uuid: "e8e05e97036343a8be26207ae1fd6034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e05e97036343a8be26207ae1fd6034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33693 } } }
I20260926 22:52:50.463754 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:50.458718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53915 (local address 127.0.0.1:33693)
0926 22:52:50.458880 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:50.458888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:50.458912 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:52:50.459443 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:50.459452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:50.459515 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:50.459652 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:50.459859 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:50.460742 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:50.461865 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:50.462352 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:50.462446 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:50.463103 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:50.463108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:50.463108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:50.463300 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:50.463344 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:50.463358 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:50.463414 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:50.463496 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 22:52:50.463565 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260926 22:52:50.463769 2133 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:50.463915 2132 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8e05e97036343a8be26207ae1fd6034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e05e97036343a8be26207ae1fd6034" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33693 } } }
I20260926 22:52:50.463989 2132 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:50.464234 2135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:50.464568 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:50.464749 2135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:50.466248 2135 catalog_manager.cc:1347] Generated new cluster ID: 6e3eb8f167a5489d9a375ffb39455d3f
I20260926 22:52:50.466292 2135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:50.468442 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:50.465970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34361 (local address 127.0.0.1:33693)
0926 22:52:50.465998 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:50.465999 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:50.466089 (+ 90us) server_negotiation.cc:407] Connection header received
0926 22:52:50.466141 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:50.466142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:50.466151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:50.466190 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:50.466308 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:50.466785 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:50.467427 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:50.467651 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:50.467706 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:50.468235 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:50.468236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:50.468236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:50.468240 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:50.468261 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:50.468265 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:50.468308 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:50.468359 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 22:52:50.468377 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":14}
I20260926 22:52:50.473419 2135 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:50.473939 2135 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:50.482137 2135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8e05e97036343a8be26207ae1fd6034: Generated new TSK 0
I20260926 22:52:50.482295 2135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:50.486557 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2058
2026-09-26T22:52:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:50 dist-test-slave-98hb krb5kdc[2157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:50 dist-test-slave-98hb krb5kdc[2157](info): set up 2 sockets
Sep 26 22:52:50 dist-test-slave-98hb krb5kdc[2157](info): commencing operation
krb5kdc: starting...
W20260926 22:52:51.437942 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:51 dist-test-slave-98hb krb5kdc[2157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:51.546814 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:51.624264 2170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:51.624477 2170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:51.624506 2170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:51.625650 2170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:51.625695 2170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:51.625712 2170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:51.625762 2170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:51.625782 2170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:51.627260 2170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:51.627497 2170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:51.627710 2170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:51.632218 2178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:51.632315 2170 server_base.cc:1034] running on GCE node
W20260926 22:52:51.632279 2176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:51.632218 2175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:51.632676 2170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:51.632958 2170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:51.634090 2170 hybrid_clock.cc:648] HybridClock initialized: now 1790463171634078 us; error 28 us; skew 500 ppm
Sep 26 22:52:51 dist-test-slave-98hb krb5kdc[2157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:51.636534 2170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:51.636977 2170 webserver.cc:460] Webserver started at http://127.0.0.1:44843/ using document root <none> and password file <none>
I20260926 22:52:51.637169 2170 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:51.637217 2170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:51.637341 2170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:51.638417 2170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "92cf5025ea714de4902f2944336a31c1"
format_stamp: "Formatted at 2026-09-26 22:52:51 on dist-test-slave-98hb"
server_key: "e327ded7f62b42ea519320b83696cecf"
server_key_iv: "e8351ee6a2bbfe1da5bba3f1c1151fb4"
server_key_version: "encryptionkey@0"
I20260926 22:52:51.638739 2170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "92cf5025ea714de4902f2944336a31c1"
format_stamp: "Formatted at 2026-09-26 22:52:51 on dist-test-slave-98hb"
server_key: "e327ded7f62b42ea519320b83696cecf"
server_key_iv: "e8351ee6a2bbfe1da5bba3f1c1151fb4"
server_key_version: "encryptionkey@0"
I20260926 22:52:51.639919 2170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:51.640591 2185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:51.640738 2170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:51.640825 2170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "92cf5025ea714de4902f2944336a31c1"
format_stamp: "Formatted at 2026-09-26 22:52:51 on dist-test-slave-98hb"
server_key: "e327ded7f62b42ea519320b83696cecf"
server_key_iv: "e8351ee6a2bbfe1da5bba3f1c1151fb4"
server_key_version: "encryptionkey@0"
I20260926 22:52:51.640887 2170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:51.644109 2170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:51.645936 2170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:51.646056 2170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:51.649631 2170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36805
I20260926 22:52:51.649662 2237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36805 every 8 connection(s)
I20260926 22:52:51.649973 2170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:51.650734 2238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:51.650799 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2170
I20260926 22:52:51.650889 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:51.651062 13904 external_mini_cluster.cc:1428] Setting key c90df4fddc0168c07bb90a921cbce4e5
I20260926 22:52:51.653329 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:51.651517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53851 (local address 127.0.0.1:36805)
0926 22:52:51.651695 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:51.651705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:51.651737 (+ 32us) server_negotiation.cc:407] Connection header received
0926 22:52:51.652203 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:51.652213 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:51.652249 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:51.652399 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:51.652937 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:51.652940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:51.652941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:51.652945 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:51.652986 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:51.652999 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:51.653073 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:51.653111 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 22:52:51.653149 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260926 22:52:51.653986 2238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1: Bootstrap starting.
I20260926 22:52:51.654263 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:51.654615 2238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:51.654862 2238 log.cc:824] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:51.655465 2238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1: No bootstrap required, opened a new log
I20260926 22:52:51.656428 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:51.655530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56747 (local address 127.0.0.1:36805)
0926 22:52:51.655564 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:51.655566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:51.655716 (+ 150us) server_negotiation.cc:407] Connection header received
0926 22:52:51.655749 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:51.655750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:51.655755 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:51.655790 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:51.656264 (+ 474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:51.656265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:51.656265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:51.656268 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:51.656279 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:51.656282 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:51.656315 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:51.656368 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 22:52:51.656377 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:52:51.657505 2238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf5025ea714de4902f2944336a31c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260926 22:52:51.657639 2238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:51.657670 2238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cf5025ea714de4902f2944336a31c1, State: Initialized, Role: FOLLOWER
I20260926 22:52:51.657809 2238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf5025ea714de4902f2944336a31c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260926 22:52:51.657886 2238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:51.657915 2238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:51.657963 2238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:51.658481 2238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf5025ea714de4902f2944336a31c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260926 22:52:51.658568 2238 leader_election.cc:302] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92cf5025ea714de4902f2944336a31c1; no voters:
I20260926 22:52:51.658707 2238 leader_election.cc:288] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:51.658761 2248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:51.658911 2248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [term 1 LEADER]: Becoming Leader. State: Replica: 92cf5025ea714de4902f2944336a31c1, State: Running, Role: LEADER
I20260926 22:52:51.658931 2238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:51.658994 2248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf5025ea714de4902f2944336a31c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260926 22:52:51.659341 2250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92cf5025ea714de4902f2944336a31c1. Latest consensus state: current_term: 1 leader_uuid: "92cf5025ea714de4902f2944336a31c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf5025ea714de4902f2944336a31c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } } }
I20260926 22:52:51.659334 2249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92cf5025ea714de4902f2944336a31c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cf5025ea714de4902f2944336a31c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } } }
I20260926 22:52:51.659437 2250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:51.659443 2249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:51.659703 2257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:51.660175 2257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:51.661564 2257 catalog_manager.cc:1347] Generated new cluster ID: 52cf9b52df394b0e90e9b04f8ac7cfb3
I20260926 22:52:51.661618 2257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:51.666630 2257 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:51.666762 2257 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:51.672561 2257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92cf5025ea714de4902f2944336a31c1: Generated new TSK 0
I20260926 22:52:51.672722 2257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:51.681794 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2170
2026-09-26T22:52:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1195 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:51 dist-test-slave-98hb krb5kdc[2269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:51 dist-test-slave-98hb krb5kdc[2269](info): set up 2 sockets
Sep 26 22:52:51 dist-test-slave-98hb krb5kdc[2269](info): commencing operation
krb5kdc: starting...
W20260926 22:52:52.629825 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:52 dist-test-slave-98hb krb5kdc[2269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:52.736639 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:52.813455 2282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:52.813654 2282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:52.813675 2282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:52.814817 2282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:52.814859 2282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:52.814872 2282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:52.814884 2282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:52.814906 2282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:52.816354 2282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:52.816567 2282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:52.816778 2282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:52.821237 2290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:52.821323 2287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:52.821249 2288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:52.821350 2282 server_base.cc:1034] running on GCE node
I20260926 22:52:52.821655 2282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:52.821959 2282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:52.823110 2282 hybrid_clock.cc:648] HybridClock initialized: now 1790463172823088 us; error 30 us; skew 500 ppm
Sep 26 22:52:52 dist-test-slave-98hb krb5kdc[2269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:52.825417 2282 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:52.825860 2282 webserver.cc:460] Webserver started at http://127.0.0.1:38533/ using document root <none> and password file <none>
I20260926 22:52:52.826092 2282 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:52.826144 2282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:52.826259 2282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:52.827220 2282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "e8f9557c8ddc42808085724cb6e07676"
format_stamp: "Formatted at 2026-09-26 22:52:52 on dist-test-slave-98hb"
server_key: "37df038a13d0119dc7536f213a358005"
server_key_iv: "f8f2eb94812a9feaea7323fa4497702d"
server_key_version: "encryptionkey@0"
I20260926 22:52:52.827525 2282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "e8f9557c8ddc42808085724cb6e07676"
format_stamp: "Formatted at 2026-09-26 22:52:52 on dist-test-slave-98hb"
server_key: "37df038a13d0119dc7536f213a358005"
server_key_iv: "f8f2eb94812a9feaea7323fa4497702d"
server_key_version: "encryptionkey@0"
I20260926 22:52:52.828707 2282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:52:52.829373 2297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:52.829521 2282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:52.829610 2282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "e8f9557c8ddc42808085724cb6e07676"
format_stamp: "Formatted at 2026-09-26 22:52:52 on dist-test-slave-98hb"
server_key: "37df038a13d0119dc7536f213a358005"
server_key_iv: "f8f2eb94812a9feaea7323fa4497702d"
server_key_version: "encryptionkey@0"
I20260926 22:52:52.829672 2282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:52.837288 2282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:52.839250 2282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:52.839375 2282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:52.842967 2282 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37743
I20260926 22:52:52.842994 2349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37743 every 8 connection(s)
I20260926 22:52:52.843287 2282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:52.844010 2350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:52.846138 2350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676: Bootstrap starting.
I20260926 22:52:52.846640 2350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:52.846879 2350 log.cc:824] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:52.847483 2350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676: No bootstrap required, opened a new log
I20260926 22:52:52.848984 2350 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9557c8ddc42808085724cb6e07676" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260926 22:52:52.849133 2350 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:52.849157 2350 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f9557c8ddc42808085724cb6e07676, State: Initialized, Role: FOLLOWER
I20260926 22:52:52.849277 2350 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9557c8ddc42808085724cb6e07676" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260926 22:52:52.849334 2350 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:52.849354 2350 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:52.849395 2350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:52.849905 2350 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9557c8ddc42808085724cb6e07676" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260926 22:52:52.849987 2350 leader_election.cc:302] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f9557c8ddc42808085724cb6e07676; no voters:
I20260926 22:52:52.850126 2350 leader_election.cc:288] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:52.850163 2353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:52.850302 2353 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [term 1 LEADER]: Becoming Leader. State: Replica: e8f9557c8ddc42808085724cb6e07676, State: Running, Role: LEADER
I20260926 22:52:52.850348 2350 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:52.850389 2353 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9557c8ddc42808085724cb6e07676" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } }
I20260926 22:52:52.850786 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2282
I20260926 22:52:52.850895 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:52.851071 13904 external_mini_cluster.cc:1428] Setting key 1df529a039fa3bb7ed79450b101faa2f
I20260926 22:52:52.851140 2354 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f9557c8ddc42808085724cb6e07676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9557c8ddc42808085724cb6e07676" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } } }
I20260926 22:52:52.851212 2354 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:52.851379 2355 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f9557c8ddc42808085724cb6e07676. Latest consensus state: current_term: 1 leader_uuid: "e8f9557c8ddc42808085724cb6e07676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f9557c8ddc42808085724cb6e07676" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37743 } } }
I20260926 22:52:52.851409 2363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:52.851436 2355 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:52.852207 2363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 22:52:52 dist-test-slave-98hb krb5kdc[2269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 22:52:52.854616 2363 catalog_manager.cc:1347] Generated new cluster ID: 7b12e274d72c41c69cb13b87e1dcc717
I20260926 22:52:52.857898 2363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:52.861701 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:52.851767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:47809 (local address 127.0.0.1:37743)
0926 22:52:52.852101 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:52.852110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:52.852141 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:52:52.852262 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:52.852269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:52.852362 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:52.852518 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:52.853272 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:52.854630 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:52.855358 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:52.855963 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:52.856102 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:52.858267 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:52.858271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:52.858272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:52.858283 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:52.860120 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:52.860525 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:52.860528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:52.860529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:52.860654 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:52.860855 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:52.860856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:52.860856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:52.861151 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:52.861307 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:52.861381 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 22:52:52.861475 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":170,"threads_started":1}
I20260926 22:52:52.862555 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:52.865970 2363 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:52.866621 2363 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:52.869680 2371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:52.863890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:52983 (local address 127.0.0.1:37743)
0926 22:52:52.863923 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:52.863925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:52.864033 (+ 108us) server_negotiation.cc:407] Connection header received
0926 22:52:52.864082 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:52.864083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:52.864093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:52.864147 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:52.864685 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:52.865428 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:52.866300 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:52.866500 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:52.866554 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:52.867552 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:52.867553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:52.867553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:52.867560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:52.868698 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:52.869019 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:52.869019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:52.869020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:52.869066 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:52.869256 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:52.869257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:52.869257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:52.869347 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:52.869444 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:52.869549 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 22:52:52.869602 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:52:52.881503 2363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f9557c8ddc42808085724cb6e07676: Generated new TSK 0
I20260926 22:52:52.881682 2363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:52.888211 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2282
W20260926 22:52:52.891989 2267 connection.cc:535] client connection to 127.0.0.1:37743 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37743): Connection reset by peer (error 104)
2026-09-26T22:52:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:52 dist-test-slave-98hb krb5kdc[2381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:52 dist-test-slave-98hb krb5kdc[2381](info): set up 2 sockets
Sep 26 22:52:52 dist-test-slave-98hb krb5kdc[2381](info): commencing operation
krb5kdc: starting...
W20260926 22:52:53.838043 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:53 dist-test-slave-98hb krb5kdc[2381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:53.944274 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:54.021104 2394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:54.021309 2394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:54.021338 2394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:54.022490 2394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:54.022531 2394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:54.022549 2394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:54.022565 2394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:54.022589 2394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:54.024027 2394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:54.024262 2394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:54.024459 2394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:54.028952 2399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:54.028952 2400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:54.028988 2402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:54.029167 2394 server_base.cc:1034] running on GCE node
I20260926 22:52:54.029356 2394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:54.029597 2394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:54.030730 2394 hybrid_clock.cc:648] HybridClock initialized: now 1790463174030718 us; error 28 us; skew 500 ppm
Sep 26 22:52:54 dist-test-slave-98hb krb5kdc[2381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:54.032889 2394 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:54.033329 2394 webserver.cc:460] Webserver started at http://127.0.0.1:35927/ using document root <none> and password file <none>
I20260926 22:52:54.033524 2394 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:54.033572 2394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:54.033692 2394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:54.034663 2394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "6ed1c78befde416193588f9670df245f"
format_stamp: "Formatted at 2026-09-26 22:52:54 on dist-test-slave-98hb"
server_key: "46d4688c79c49085efd87cbde31f6fce"
server_key_iv: "7e41c6fcaf3b46f7fd18a44fde767da5"
server_key_version: "encryptionkey@0"
I20260926 22:52:54.034991 2394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "6ed1c78befde416193588f9670df245f"
format_stamp: "Formatted at 2026-09-26 22:52:54 on dist-test-slave-98hb"
server_key: "46d4688c79c49085efd87cbde31f6fce"
server_key_iv: "7e41c6fcaf3b46f7fd18a44fde767da5"
server_key_version: "encryptionkey@0"
I20260926 22:52:54.036168 2394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 22:52:54.036783 2409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:54.036926 2394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:52:54.037001 2394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "6ed1c78befde416193588f9670df245f"
format_stamp: "Formatted at 2026-09-26 22:52:54 on dist-test-slave-98hb"
server_key: "46d4688c79c49085efd87cbde31f6fce"
server_key_iv: "7e41c6fcaf3b46f7fd18a44fde767da5"
server_key_version: "encryptionkey@0"
I20260926 22:52:54.037060 2394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:54.049955 2394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:54.051643 2394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:54.051779 2394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:54.055588 2394 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44349
I20260926 22:52:54.055619 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44349 every 8 connection(s)
I20260926 22:52:54.055948 2394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:54.056797 2462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:54.057931 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2394
I20260926 22:52:54.058038 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:54.058247 13904 external_mini_cluster.cc:1428] Setting key 6cfe42a653eebaafc5f25697c93545e4
I20260926 22:52:54.059552 2462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f: Bootstrap starting.
I20260926 22:52:54.060240 2462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:54.060534 2462 log.cc:824] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:54.061237 2462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f: No bootstrap required, opened a new log
I20260926 22:52:54.063427 2462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed1c78befde416193588f9670df245f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260926 22:52:54.063601 2462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:54.063647 2462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed1c78befde416193588f9670df245f, State: Initialized, Role: FOLLOWER
I20260926 22:52:54.063789 2462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed1c78befde416193588f9670df245f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260926 22:52:54.063911 2462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:54.063956 2462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:54.064009 2462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:54.064090 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:54.058766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:54875 (local address 127.0.0.1:44349)
0926 22:52:54.058954 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:54.058964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:54.059000 (+ 36us) server_negotiation.cc:407] Connection header received
0926 22:52:54.059513 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:54.059522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:54.059587 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:54.059732 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:54.059930 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:54.061199 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:54.062023 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:54.062657 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:54.062777 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:54.063402 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:54.063405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:54.063406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:54.063600 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:54.063638 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:54.063654 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:54.063736 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:54.063839 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 22:52:54.063866 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260926 22:52:54.064597 2462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed1c78befde416193588f9670df245f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260926 22:52:54.064697 2462 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed1c78befde416193588f9670df245f; no voters:
I20260926 22:52:54.064867 2462 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:54.064906 2467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:54.065030 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:54.065057 2467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [term 1 LEADER]: Becoming Leader. State: Replica: 6ed1c78befde416193588f9670df245f, State: Running, Role: LEADER
I20260926 22:52:54.065104 2462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:54.065163 2467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed1c78befde416193588f9670df245f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260926 22:52:54.065446 2467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed1c78befde416193588f9670df245f. Latest consensus state: current_term: 1 leader_uuid: "6ed1c78befde416193588f9670df245f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed1c78befde416193588f9670df245f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } } }
I20260926 22:52:54.065503 2467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:54.065629 2469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed1c78befde416193588f9670df245f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed1c78befde416193588f9670df245f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } } }
I20260926 22:52:54.065672 2469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:54.065832 2472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:54.066308 2472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:54.067829 2472 catalog_manager.cc:1347] Generated new cluster ID: 9b6e130b9a97421c9c17757a6fadbfb2
I20260926 22:52:54.067879 2472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:54.069557 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:54.066531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:59719 (local address 127.0.0.1:44349)
0926 22:52:54.066565 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:54.066567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:54.066861 (+ 294us) server_negotiation.cc:407] Connection header received
0926 22:52:54.066935 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:54.066937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:54.066946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:54.066974 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:54.067154 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:54.067781 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:54.068488 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:54.068703 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:54.068754 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:54.069328 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:54.069329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:54.069329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:54.069333 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:54.069347 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:54.069351 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:54.069398 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:54.069466 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 22:52:54.069475 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:52:54.094769 2472 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:54.095501 2472 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:54.109472 2472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed1c78befde416193588f9670df245f: Generated new TSK 0
I20260926 22:52:54.109706 2472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:54.115996 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2394
W20260926 22:52:54.120621 2379 connection.cc:535] client connection to 127.0.0.1:44349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44349): Connection reset by peer (error 104)
2026-09-26T22:52:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:54 dist-test-slave-98hb krb5kdc[2493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:54 dist-test-slave-98hb krb5kdc[2493](info): set up 2 sockets
Sep 26 22:52:54 dist-test-slave-98hb krb5kdc[2493](info): commencing operation
krb5kdc: starting...
W20260926 22:52:55.053943 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:55 dist-test-slave-98hb krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:55.162488 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33373
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:55.242347 2506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:55.242547 2506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:55.242575 2506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:55.243738 2506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:55.243783 2506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:55.243803 2506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:55.243818 2506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:55.243834 2506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:55.245262 2506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:55.245500 2506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:55.245697 2506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:55.250131 2512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:55.250144 2514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:55.250113 2511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:55.250238 2506 server_base.cc:1034] running on GCE node
I20260926 22:52:55.250622 2506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:55.250883 2506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:55.252036 2506 hybrid_clock.cc:648] HybridClock initialized: now 1790463175252021 us; error 27 us; skew 500 ppm
Sep 26 22:52:55 dist-test-slave-98hb krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:55.258627 2506 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:55.259061 2506 webserver.cc:460] Webserver started at http://127.0.0.1:42561/ using document root <none> and password file <none>
I20260926 22:52:55.259267 2506 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:55.259315 2506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:55.259436 2506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:55.260440 2506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "f71eb368ead14e6ebb52a701d735cf27"
format_stamp: "Formatted at 2026-09-26 22:52:55 on dist-test-slave-98hb"
server_key: "af24e58960e2bcc7dcb6f8636b14e0da"
server_key_iv: "44f0684a6e64f9b7d4ec95c5c1396cd1"
server_key_version: "encryptionkey@0"
I20260926 22:52:55.260768 2506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "f71eb368ead14e6ebb52a701d735cf27"
format_stamp: "Formatted at 2026-09-26 22:52:55 on dist-test-slave-98hb"
server_key: "af24e58960e2bcc7dcb6f8636b14e0da"
server_key_iv: "44f0684a6e64f9b7d4ec95c5c1396cd1"
server_key_version: "encryptionkey@0"
I20260926 22:52:55.262005 2506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:55.262641 2521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:55.262791 2506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:55.262873 2506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "f71eb368ead14e6ebb52a701d735cf27"
format_stamp: "Formatted at 2026-09-26 22:52:55 on dist-test-slave-98hb"
server_key: "af24e58960e2bcc7dcb6f8636b14e0da"
server_key_iv: "44f0684a6e64f9b7d4ec95c5c1396cd1"
server_key_version: "encryptionkey@0"
I20260926 22:52:55.262933 2506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:55.266072 2506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:55.267696 2506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:55.267827 2506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:55.271399 2506 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33373
I20260926 22:52:55.271427 2573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33373 every 8 connection(s)
I20260926 22:52:55.271711 2506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:55.272531 2574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:55.274706 2574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27: Bootstrap starting.
I20260926 22:52:55.275233 2574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:55.275468 2574 log.cc:824] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:55.276093 2574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27: No bootstrap required, opened a new log
I20260926 22:52:55.276511 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2506
I20260926 22:52:55.276605 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:55.276777 13904 external_mini_cluster.cc:1428] Setting key 850ecfa34ac896edf69cd249413ecaf0
I20260926 22:52:55.277935 2574 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71eb368ead14e6ebb52a701d735cf27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33373 } }
I20260926 22:52:55.278088 2574 raft_consensus.cc:374] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:55.278158 2574 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71eb368ead14e6ebb52a701d735cf27, State: Initialized, Role: FOLLOWER
I20260926 22:52:55.278293 2574 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71eb368ead14e6ebb52a701d735cf27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33373 } }
I20260926 22:52:55.278388 2574 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:55.278434 2574 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:55.278475 2574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:55.279148 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:55.277202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:45329 (local address 127.0.0.1:33373)
0926 22:52:55.277370 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:55.277378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:55.277402 (+ 24us) server_negotiation.cc:407] Connection header received
0926 22:52:55.277899 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:55.277905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:55.277930 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:55.278038 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:55.278731 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:55.278735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:55.278735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:55.278738 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:55.278781 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:55.278792 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:55.278844 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:55.278921 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 22:52:55.278956 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260926 22:52:55.279418 2574 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71eb368ead14e6ebb52a701d735cf27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33373 } }
I20260926 22:52:55.279510 2574 leader_election.cc:302] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71eb368ead14e6ebb52a701d735cf27; no voters:
I20260926 22:52:55.279651 2574 leader_election.cc:288] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:55.279671 2579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:55.279814 2579 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [term 1 LEADER]: Becoming Leader. State: Replica: f71eb368ead14e6ebb52a701d735cf27, State: Running, Role: LEADER
I20260926 22:52:55.279922 2574 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:55.279939 2579 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71eb368ead14e6ebb52a701d735cf27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33373 } }
I20260926 22:52:55.279989 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:55.280267 2579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71eb368ead14e6ebb52a701d735cf27. Latest consensus state: current_term: 1 leader_uuid: "f71eb368ead14e6ebb52a701d735cf27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71eb368ead14e6ebb52a701d735cf27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33373 } } }
I20260926 22:52:55.280483 2579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:55.280419 2581 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f71eb368ead14e6ebb52a701d735cf27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71eb368ead14e6ebb52a701d735cf27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33373 } } }
I20260926 22:52:55.280857 2581 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27 [sys.catalog]: This master's current role is: LEADER
W20260926 22:52:55.281601 2598 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:52:55.281646 2598 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:52:55.281687 2588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:55.281857 2588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:55.282488 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:55.281400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:59159 (local address 127.0.0.1:33373)
0926 22:52:55.281435 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:55.281437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:55.281582 (+ 145us) server_negotiation.cc:407] Connection header received
0926 22:52:55.281674 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:55.281675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:55.281679 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:55.281704 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:55.282286 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:55.282288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:55.282288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:55.282292 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:55.282305 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:55.282309 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:55.282336 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:55.282421 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 22:52:55.282436 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:52:55.283242 2588 catalog_manager.cc:1347] Generated new cluster ID: 7422160bb325443d90fd12701d56e7c5
I20260926 22:52:55.283298 2588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:55.287132 2588 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:55.287272 2588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:55.293640 2588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f71eb368ead14e6ebb52a701d735cf27: Generated new TSK 0
I20260926 22:52:55.293810 2588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:55.298431 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2506
2026-09-26T22:52:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1181 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:55 dist-test-slave-98hb krb5kdc[2605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:55 dist-test-slave-98hb krb5kdc[2605](info): set up 2 sockets
Sep 26 22:52:55 dist-test-slave-98hb krb5kdc[2605](info): commencing operation
krb5kdc: starting...
W20260926 22:52:56.241900 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:56 dist-test-slave-98hb krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:56.348388 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:56.425634 2618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:56.425864 2618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:56.425889 2618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:56.427011 2618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:56.427053 2618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:56.427075 2618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:56.427100 2618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:56.427122 2618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:56.428546 2618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:56.428756 2618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:56.428949 2618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:56.433333 2626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:56.433352 2624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:56.433323 2623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:56.433537 2618 server_base.cc:1034] running on GCE node
I20260926 22:52:56.433863 2618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:56.434118 2618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:56.435256 2618 hybrid_clock.cc:648] HybridClock initialized: now 1790463176435239 us; error 29 us; skew 500 ppm
Sep 26 22:52:56 dist-test-slave-98hb krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:56.437471 2618 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:56.437888 2618 webserver.cc:460] Webserver started at http://127.0.0.1:36119/ using document root <none> and password file <none>
I20260926 22:52:56.438076 2618 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:56.438117 2618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:56.438231 2618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:56.439193 2618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "df0970cdc4ef422c8522f9fe3fa74a06"
format_stamp: "Formatted at 2026-09-26 22:52:56 on dist-test-slave-98hb"
server_key: "93772d6ce986b1df80398760811e4d51"
server_key_iv: "dfab1b82e8d05b18640811bdaf3a9615"
server_key_version: "encryptionkey@0"
I20260926 22:52:56.439502 2618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "df0970cdc4ef422c8522f9fe3fa74a06"
format_stamp: "Formatted at 2026-09-26 22:52:56 on dist-test-slave-98hb"
server_key: "93772d6ce986b1df80398760811e4d51"
server_key_iv: "dfab1b82e8d05b18640811bdaf3a9615"
server_key_version: "encryptionkey@0"
I20260926 22:52:56.440634 2618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:52:56.441226 2633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:56.441354 2618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:56.441437 2618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "df0970cdc4ef422c8522f9fe3fa74a06"
format_stamp: "Formatted at 2026-09-26 22:52:56 on dist-test-slave-98hb"
server_key: "93772d6ce986b1df80398760811e4d51"
server_key_iv: "dfab1b82e8d05b18640811bdaf3a9615"
server_key_version: "encryptionkey@0"
I20260926 22:52:56.441498 2618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:56.451540 2618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:56.453068 2618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:56.453191 2618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:56.456658 2618 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40055
I20260926 22:52:56.456705 2685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40055 every 8 connection(s)
I20260926 22:52:56.457017 2618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:56.457866 2686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:56.459949 2686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06: Bootstrap starting.
I20260926 22:52:56.460469 2686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:56.460724 2686 log.cc:824] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:56.461335 2686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06: No bootstrap required, opened a new log
I20260926 22:52:56.462103 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2618
I20260926 22:52:56.462208 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:56.462374 13904 external_mini_cluster.cc:1428] Setting key b95d0746c3ac9bf5aa13ad4aab34677b
I20260926 22:52:56.463236 2686 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0970cdc4ef422c8522f9fe3fa74a06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40055 } }
I20260926 22:52:56.463393 2686 raft_consensus.cc:374] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:56.463428 2686 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0970cdc4ef422c8522f9fe3fa74a06, State: Initialized, Role: FOLLOWER
I20260926 22:52:56.463582 2686 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0970cdc4ef422c8522f9fe3fa74a06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40055 } }
I20260926 22:52:56.463657 2686 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:56.463678 2686 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:56.463722 2686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:56.464727 2686 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0970cdc4ef422c8522f9fe3fa74a06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40055 } }
I20260926 22:52:56.464828 2686 leader_election.cc:302] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0970cdc4ef422c8522f9fe3fa74a06; no voters:
I20260926 22:52:56.464969 2686 leader_election.cc:288] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:56.465029 2691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:56.465155 2691 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [term 1 LEADER]: Becoming Leader. State: Replica: df0970cdc4ef422c8522f9fe3fa74a06, State: Running, Role: LEADER
I20260926 22:52:56.465183 2686 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:56.466310 2691 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0970cdc4ef422c8522f9fe3fa74a06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40055 } }
I20260926 22:52:56.466843 2693 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0970cdc4ef422c8522f9fe3fa74a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0970cdc4ef422c8522f9fe3fa74a06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40055 } } }
I20260926 22:52:56.466954 2693 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:56.467198 2705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:56.466843 2692 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0970cdc4ef422c8522f9fe3fa74a06. Latest consensus state: current_term: 1 leader_uuid: "df0970cdc4ef422c8522f9fe3fa74a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0970cdc4ef422c8522f9fe3fa74a06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40055 } } }
I20260926 22:52:56.467536 2692 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:56.467881 2705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 22:52:56 dist-test-slave-98hb krb5kdc[2605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 22:52:56.470155 2705 catalog_manager.cc:1347] Generated new cluster ID: 838553e9503344d88ab53ab14ee1be30
I20260926 22:52:56.470207 2705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:56.472999 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:56.462868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:50283 (local address 127.0.0.1:40055)
0926 22:52:56.463057 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:56.463066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:56.463100 (+ 34us) server_negotiation.cc:407] Connection header received
0926 22:52:56.463210 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:56.463217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:56.463273 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:56.463414 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:56.464055 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:56.465369 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:56.466316 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:56.466881 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:56.466980 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:56.469626 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:56.469629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:56.469630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:56.469638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:56.471415 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:56.471847 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:56.471849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:56.471849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:56.471966 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:56.472181 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:56.472181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:56.472181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:56.472450 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:56.472595 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:56.472727 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 22:52:56.472816 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260926 22:52:56.473922 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:56.484427 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:56.475302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:59715 (local address 127.0.0.1:40055)
0926 22:52:56.475333 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:56.475335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:56.477901 (+ 2566us) server_negotiation.cc:407] Connection header received
0926 22:52:56.477961 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:56.477962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:56.477971 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:56.478021 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:56.478603 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:56.479295 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:56.480278 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:56.480605 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:56.480655 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:56.481939 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:56.481940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:56.481940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:52:56.481946 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:56.483361 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:56.483744 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:56.483744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:56.483745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:56.483790 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:52:56.483983 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:52:56.483983 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:52:56.483983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:52:56.484082 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:56.484172 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:56.484289 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 22:52:56.484345 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:52:56.499641 2705 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:56.500156 2705 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:56.504722 2705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0970cdc4ef422c8522f9fe3fa74a06: Generated new TSK 0
I20260926 22:52:56.504884 2705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:56.509493 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2618
W20260926 22:52:56.513464 2603 connection.cc:535] client connection to 127.0.0.1:40055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40055): Connection reset by peer (error 104)
2026-09-26T22:52:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:56 dist-test-slave-98hb krb5kdc[2717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:56 dist-test-slave-98hb krb5kdc[2717](info): set up 2 sockets
Sep 26 22:52:56 dist-test-slave-98hb krb5kdc[2717](info): commencing operation
krb5kdc: starting...
W20260926 22:52:57.449997 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:57 dist-test-slave-98hb krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:57.556452 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:57.634119 2730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:57.634310 2730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:57.634330 2730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:57.635440 2730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:57.635480 2730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:57.635492 2730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:57.635510 2730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:57.635535 2730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:57.636969 2730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43447
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:57.637190 2730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:57.637387 2730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:57.641813 2736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:57.641808 2735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:57.641870 2730 server_base.cc:1034] running on GCE node
W20260926 22:52:57.641805 2738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:57.642278 2730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:57.642540 2730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:57.643682 2730 hybrid_clock.cc:648] HybridClock initialized: now 1790463177643670 us; error 24 us; skew 500 ppm
Sep 26 22:52:57 dist-test-slave-98hb krb5kdc[2717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:57.645963 2730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:57.646386 2730 webserver.cc:460] Webserver started at http://127.0.0.1:38837/ using document root <none> and password file <none>
I20260926 22:52:57.646593 2730 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:57.646718 2730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:57.646827 2730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:57.647754 2730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "88bf715fba99479e86b1179d1a0f3c1e"
format_stamp: "Formatted at 2026-09-26 22:52:57 on dist-test-slave-98hb"
server_key: "dc79f17c189a3ef16dee88f7004ec765"
server_key_iv: "e3ef879d758f6a79029d656343f52df1"
server_key_version: "encryptionkey@0"
I20260926 22:52:57.648026 2730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "88bf715fba99479e86b1179d1a0f3c1e"
format_stamp: "Formatted at 2026-09-26 22:52:57 on dist-test-slave-98hb"
server_key: "dc79f17c189a3ef16dee88f7004ec765"
server_key_iv: "e3ef879d758f6a79029d656343f52df1"
server_key_version: "encryptionkey@0"
I20260926 22:52:57.649127 2730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:57.649718 2745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:57.649883 2730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:57.649976 2730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "88bf715fba99479e86b1179d1a0f3c1e"
format_stamp: "Formatted at 2026-09-26 22:52:57 on dist-test-slave-98hb"
server_key: "dc79f17c189a3ef16dee88f7004ec765"
server_key_iv: "e3ef879d758f6a79029d656343f52df1"
server_key_version: "encryptionkey@0"
I20260926 22:52:57.650043 2730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:57.670172 2730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:57.671770 2730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:57.671890 2730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:57.675428 2730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43447
I20260926 22:52:57.675462 2797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43447 every 8 connection(s)
I20260926 22:52:57.675766 2730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:57.676683 2798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:57.678862 2798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e: Bootstrap starting.
I20260926 22:52:57.679390 2798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:57.679636 2798 log.cc:824] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:57.680255 2798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e: No bootstrap required, opened a new log
I20260926 22:52:57.680444 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2730
I20260926 22:52:57.680565 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:57.680761 13904 external_mini_cluster.cc:1428] Setting key f653db5632b014db47c4a2dd2a64ed4f
I20260926 22:52:57.682291 2798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bf715fba99479e86b1179d1a0f3c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } }
I20260926 22:52:57.682423 2798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:57.682444 2798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88bf715fba99479e86b1179d1a0f3c1e, State: Initialized, Role: FOLLOWER
I20260926 22:52:57.682545 2798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bf715fba99479e86b1179d1a0f3c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } }
I20260926 22:52:57.682610 2798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:57.682638 2798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:57.682713 2798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:57.683254 2798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bf715fba99479e86b1179d1a0f3c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } }
I20260926 22:52:57.683336 2798 leader_election.cc:302] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88bf715fba99479e86b1179d1a0f3c1e; no voters:
I20260926 22:52:57.683473 2798 leader_election.cc:288] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:57.683521 2803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:57.683657 2803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [term 1 LEADER]: Becoming Leader. State: Replica: 88bf715fba99479e86b1179d1a0f3c1e, State: Running, Role: LEADER
I20260926 22:52:57.683715 2798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:57.683760 2803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bf715fba99479e86b1179d1a0f3c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } }
I20260926 22:52:57.684080 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88bf715fba99479e86b1179d1a0f3c1e. Latest consensus state: current_term: 1 leader_uuid: "88bf715fba99479e86b1179d1a0f3c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bf715fba99479e86b1179d1a0f3c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } } }
I20260926 22:52:57.684206 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:57.684438 2804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88bf715fba99479e86b1179d1a0f3c1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88bf715fba99479e86b1179d1a0f3c1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43447 } } }
I20260926 22:52:57.684556 2804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:57.684850 2811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:57.685413 2811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:57.686959 2811 catalog_manager.cc:1347] Generated new cluster ID: 26a6cbffbced44008808b5373668610d
I20260926 22:52:57.687038 2811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:57.687158 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:57.681217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:45227 (local address 127.0.0.1:43447)
0926 22:52:57.681368 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:57.681375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:57.681401 (+ 26us) server_negotiation.cc:407] Connection header received
0926 22:52:57.682424 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:57.682434 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:57.682502 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:57.682649 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:57.682855 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:57.683851 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:57.684922 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:57.685443 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:57.685541 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:57.686424 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:57.686427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:57.686428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:57.686633 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:57.686669 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:57.686684 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:57.686765 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:57.686896 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 22:52:57.686923 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260926 22:52:57.688019 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:57.692636 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:57.690002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:46489 (local address 127.0.0.1:43447)
0926 22:52:57.690034 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:57.690036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:57.690230 (+ 194us) server_negotiation.cc:407] Connection header received
0926 22:52:57.690284 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:57.690285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:57.690294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:57.690338 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:57.690463 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:57.690911 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:57.691582 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:52:57.691790 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:52:57.691841 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:52:57.692416 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:57.692418 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:57.692418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:57.692422 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:57.692435 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:57.692439 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:57.692482 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:57.692548 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 22:52:57.692556 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 22:52:57.712571 2811 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:57.713074 2811 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:57.716455 2811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88bf715fba99479e86b1179d1a0f3c1e: Generated new TSK 0
I20260926 22:52:57.716619 2811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:57.727162 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2730
W20260926 22:52:57.730998 2715 connection.cc:535] client connection to 127.0.0.1:43447 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43447): Connection reset by peer (error 104)
2026-09-26T22:52:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:57 dist-test-slave-98hb krb5kdc[2829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:57 dist-test-slave-98hb krb5kdc[2829](info): set up 2 sockets
Sep 26 22:52:57 dist-test-slave-98hb krb5kdc[2829](info): commencing operation
krb5kdc: starting...
W20260926 22:52:58.681870 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:58 dist-test-slave-98hb krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 22:52:58.788431 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:52:58.865252 2842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:52:58.865440 2842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:52:58.865460 2842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:52:58.866578 2842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:52:58.866618 2842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:52:58.866631 2842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:52:58.866654 2842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:52:58.866669 2842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:52:58.868078 2842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:52:58.868295 2842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:52:58.868486 2842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:52:58.872838 2850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:58.872853 2842 server_base.cc:1034] running on GCE node
W20260926 22:52:58.872895 2847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:52:58.872840 2848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:52:58.873327 2842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:52:58.873596 2842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:52:58.874724 2842 hybrid_clock.cc:648] HybridClock initialized: now 1790463178874698 us; error 38 us; skew 500 ppm
Sep 26 22:52:58 dist-test-slave-98hb krb5kdc[2829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:52:58.876935 2842 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 22:52:58.877312 2842 webserver.cc:460] Webserver started at http://127.0.0.1:45109/ using document root <none> and password file <none>
I20260926 22:52:58.877497 2842 fs_manager.cc:359] Metadata directory not provided
I20260926 22:52:58.877542 2842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:52:58.877663 2842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:52:58.878648 2842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "32213a94e2c34decae054856364407e6"
format_stamp: "Formatted at 2026-09-26 22:52:58 on dist-test-slave-98hb"
server_key: "1d7fd44221ca06945dfdbb16632bcb53"
server_key_iv: "182b90b8ad0a2faafa609781bf06b1d9"
server_key_version: "encryptionkey@0"
I20260926 22:52:58.878969 2842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "32213a94e2c34decae054856364407e6"
format_stamp: "Formatted at 2026-09-26 22:52:58 on dist-test-slave-98hb"
server_key: "1d7fd44221ca06945dfdbb16632bcb53"
server_key_iv: "182b90b8ad0a2faafa609781bf06b1d9"
server_key_version: "encryptionkey@0"
I20260926 22:52:58.880162 2842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:52:58.880867 2857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:52:58.881033 2842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:52:58.881112 2842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "32213a94e2c34decae054856364407e6"
format_stamp: "Formatted at 2026-09-26 22:52:58 on dist-test-slave-98hb"
server_key: "1d7fd44221ca06945dfdbb16632bcb53"
server_key_iv: "182b90b8ad0a2faafa609781bf06b1d9"
server_key_version: "encryptionkey@0"
I20260926 22:52:58.881172 2842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:52:58.884267 2842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:52:58.885919 2842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:52:58.886044 2842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:52:58.889482 2842 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41835
I20260926 22:52:58.889509 2909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41835 every 8 connection(s)
I20260926 22:52:58.889822 2842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:52:58.890618 2910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:52:58.892319 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2842
I20260926 22:52:58.892423 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:52:58.892578 13904 external_mini_cluster.cc:1428] Setting key 3755fe680be02cbe77d7913c4901e179
I20260926 22:52:58.892793 2910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6: Bootstrap starting.
I20260926 22:52:58.893349 2910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6: Neither blocks nor log segments found. Creating new log.
I20260926 22:52:58.893602 2910 log.cc:824] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6: Log is configured to *not* fsync() on all Append() calls
I20260926 22:52:58.894228 2910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6: No bootstrap required, opened a new log
I20260926 22:52:58.894904 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:58.893012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:39287 (local address 127.0.0.1:41835)
0926 22:52:58.893211 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:58.893220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:58.893247 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:52:58.893727 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:58.893766 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:58.893797 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:58.893920 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:58.894530 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:58.894533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:58.894533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:58.894537 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:58.894584 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:58.894594 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:58.894671 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:58.894703 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 22:52:58.894742 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260926 22:52:58.895802 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:52:58.896286 2910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32213a94e2c34decae054856364407e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260926 22:52:58.896409 2910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:52:58.896440 2910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32213a94e2c34decae054856364407e6, State: Initialized, Role: FOLLOWER
I20260926 22:52:58.896548 2910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32213a94e2c34decae054856364407e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260926 22:52:58.896620 2910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:52:58.896651 2910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:52:58.896714 2910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:52:58.897331 2910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32213a94e2c34decae054856364407e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260926 22:52:58.897425 2910 leader_election.cc:302] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32213a94e2c34decae054856364407e6; no voters:
I20260926 22:52:58.897588 2910 leader_election.cc:288] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:52:58.897612 2920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:52:58.897751 2920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [term 1 LEADER]: Becoming Leader. State: Replica: 32213a94e2c34decae054856364407e6, State: Running, Role: LEADER
I20260926 22:52:58.897840 2910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:52:58.898169 2913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:52:58.897074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.255:35705 (local address 127.0.0.1:41835)
0926 22:52:58.897104 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 22:52:58.897106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:52:58.897219 (+ 113us) server_negotiation.cc:407] Connection header received
0926 22:52:58.897283 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:52:58.897283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:52:58.897288 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:52:58.897328 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:52:58.897903 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:52:58.897905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:52:58.897921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:52:58.897925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:52:58.897939 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:52:58.897944 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:52:58.897998 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:52:58.898099 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 22:52:58.898112 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 22:52:58.897832 2920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32213a94e2c34decae054856364407e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } }
I20260926 22:52:58.899498 2921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32213a94e2c34decae054856364407e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32213a94e2c34decae054856364407e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } } }
I20260926 22:52:58.899619 2922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32213a94e2c34decae054856364407e6. Latest consensus state: current_term: 1 leader_uuid: "32213a94e2c34decae054856364407e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32213a94e2c34decae054856364407e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41835 } } }
I20260926 22:52:58.899703 2921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:52:58.899739 2922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6 [sys.catalog]: This master's current role is: LEADER
W20260926 22:52:58.900141 2934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:52:58.900180 2934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:52:58.900311 2936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:52:58.900446 2936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:52:58.901644 2936 catalog_manager.cc:1347] Generated new cluster ID: 4ca1b219cf8e4744b446af2e34173def
I20260926 22:52:58.901700 2936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:52:58.928210 2936 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:52:58.928397 2936 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:52:58.937330 2936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32213a94e2c34decae054856364407e6: Generated new TSK 0
I20260926 22:52:58.937521 2936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:52:58.946627 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2842
2026-09-26T22:52:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:52:58 dist-test-slave-98hb krb5kdc[2941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:52:58 dist-test-slave-98hb krb5kdc[2941](info): set up 2 sockets
Sep 26 22:52:58 dist-test-slave-98hb krb5kdc[2941](info): commencing operation
krb5kdc: starting...
W20260926 22:52:59.893879 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:52:59 dist-test-slave-98hb krb5kdc[2941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:53:00.002475 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45163
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:45163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:53:00.080371 2954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:00.080569 2954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:00.080600 2954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:00.081712 2954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:00.081781 2954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:00.081801 2954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:00.081817 2954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:00.081833 2954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:00.083256 2954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:45163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:45163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:00.083494 2954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:00.083685 2954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:00.088088 2960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:00.088094 2962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:00.088205 2954 server_base.cc:1034] running on GCE node
W20260926 22:53:00.088236 2959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:00.088536 2954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:00.088820 2954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:00.089951 2954 hybrid_clock.cc:648] HybridClock initialized: now 1790463180089936 us; error 34 us; skew 500 ppm
Sep 26 22:53:00 dist-test-slave-98hb krb5kdc[2941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463180, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:00.092196 2954 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:53:00.092607 2954 webserver.cc:460] Webserver started at http://127.13.148.62:40329/ using document root <none> and password file <none>
I20260926 22:53:00.092795 2954 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:00.092841 2954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:00.092957 2954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:00.093983 2954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "4911431533ab4301b0efa8f93fd8822c"
format_stamp: "Formatted at 2026-09-26 22:53:00 on dist-test-slave-98hb"
server_key: "b3ce6fa32772b66482945166722d7def"
server_key_iv: "8847167384814a7f4b7668056bcc4274"
server_key_version: "encryptionkey@0"
I20260926 22:53:00.094300 2954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "4911431533ab4301b0efa8f93fd8822c"
format_stamp: "Formatted at 2026-09-26 22:53:00 on dist-test-slave-98hb"
server_key: "b3ce6fa32772b66482945166722d7def"
server_key_iv: "8847167384814a7f4b7668056bcc4274"
server_key_version: "encryptionkey@0"
I20260926 22:53:00.095439 2954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:53:00.096057 2969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:00.096215 2954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:00.096288 2954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "4911431533ab4301b0efa8f93fd8822c"
format_stamp: "Formatted at 2026-09-26 22:53:00 on dist-test-slave-98hb"
server_key: "b3ce6fa32772b66482945166722d7def"
server_key_iv: "8847167384814a7f4b7668056bcc4274"
server_key_version: "encryptionkey@0"
I20260926 22:53:00.096338 2954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:00.106794 2954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:00.108379 2954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:00.108479 2954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:00.112103 2954 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:45163
I20260926 22:53:00.112143 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:45163 every 8 connection(s)
I20260926 22:53:00.112464 2954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:53:00.113247 3022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:00.115437 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c: Bootstrap starting.
I20260926 22:53:00.115970 3022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:00.116233 3022 log.cc:824] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:00.116483 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 2954
I20260926 22:53:00.116605 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:53:00.116804 13904 external_mini_cluster.cc:1428] Setting key 99e445890d589c4ea8be7b4c580757c5
I20260926 22:53:00.116936 3022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c: No bootstrap required, opened a new log
I20260926 22:53:00.118994 3022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4911431533ab4301b0efa8f93fd8822c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45163 } }
I20260926 22:53:00.119164 3022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:53:00.119196 3022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4911431533ab4301b0efa8f93fd8822c, State: Initialized, Role: FOLLOWER
I20260926 22:53:00.119328 3022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4911431533ab4301b0efa8f93fd8822c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45163 } }
I20260926 22:53:00.119405 3022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:53:00.119439 3022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:53:00.119481 3022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:00.120127 3022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4911431533ab4301b0efa8f93fd8822c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45163 } }
I20260926 22:53:00.120231 3022 leader_election.cc:302] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4911431533ab4301b0efa8f93fd8822c; no voters:
I20260926 22:53:00.120366 3022 leader_election.cc:288] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:53:00.120409 3027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:53:00.120544 3027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [term 1 LEADER]: Becoming Leader. State: Replica: 4911431533ab4301b0efa8f93fd8822c, State: Running, Role: LEADER
I20260926 22:53:00.120585 3022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:00.120684 3027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4911431533ab4301b0efa8f93fd8822c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45163 } }
I20260926 22:53:00.122068 3029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4911431533ab4301b0efa8f93fd8822c. Latest consensus state: current_term: 1 leader_uuid: "4911431533ab4301b0efa8f93fd8822c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4911431533ab4301b0efa8f93fd8822c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45163 } } }
I20260926 22:53:00.121953 3028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4911431533ab4301b0efa8f93fd8822c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4911431533ab4301b0efa8f93fd8822c" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 45163 } } }
I20260926 22:53:00.122195 3028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [sys.catalog]: This master's current role is: LEADER
W20260926 22:53:00.122331 3042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 22:53:00.122370 3029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c [sys.catalog]: This master's current role is: LEADER
W20260926 22:53:00.122401 3042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:53:00.122550 3044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:53:00.122717 3044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:53:00.124100 3044 catalog_manager.cc:1347] Generated new cluster ID: ed534c535fd346fbbe33d669299ab414
I20260926 22:53:00.124150 3044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 22:53:00 dist-test-slave-98hb krb5kdc[2941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:53:00.128827 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:00.117246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57535 (local address 127.13.148.62:45163)
0926 22:53:00.117405 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:00.117415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:00.117448 (+ 33us) server_negotiation.cc:407] Connection header received
0926 22:53:00.118227 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:00.118237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:00.118308 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:00.118458 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:00.119024 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:00.120616 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:00.121697 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:00.122342 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:00.122443 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:00.125384 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:00.125389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:00.125389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:00.125401 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:00.127224 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:00.127677 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:00.127680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:00.127680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:00.127776 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:00.127991 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:00.127992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:00.127993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:00.128265 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:00.128408 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:00.128544 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 22:53:00.128638 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260926 22:53:00.129964 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:53:00.132851 3044 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:53:00.133487 3044 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:53:00.138103 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:00.131360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35871 (local address 127.13.148.62:45163)
0926 22:53:00.131401 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:00.131402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:00.131550 (+ 148us) server_negotiation.cc:407] Connection header received
0926 22:53:00.131594 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:00.131595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:00.131604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:00.131661 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:00.132214 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:00.132971 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:00.133936 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:00.134270 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:00.134337 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:00.135535 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:00.135537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:00.135537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:00.135543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:00.136981 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:00.137349 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:00.137350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:00.137351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:00.137405 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:00.137593 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:00.137593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:00.137593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:00.137710 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:00.137819 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:00.137957 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 22:53:00.138016 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 22:53:00.144594 3044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4911431533ab4301b0efa8f93fd8822c: Generated new TSK 0
I20260926 22:53:00.144766 3044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:53:00.148804 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 2954
W20260926 22:53:00.152472 2939 connection.cc:535] client connection to 127.13.148.62:45163 recv error: Network error: failed to read from TLS socket (remote: 127.13.148.62:45163): Connection reset by peer (error 104)
2026-09-26T22:53:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:53:00 dist-test-slave-98hb krb5kdc[3053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:53:00 dist-test-slave-98hb krb5kdc[3053](info): set up 2 sockets
Sep 26 22:53:00 dist-test-slave-98hb krb5kdc[3053](info): commencing operation
krb5kdc: starting...
W20260926 22:53:01.089888 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:53:01 dist-test-slave-98hb krb5kdc[3053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:53:01.198477 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39107
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:34239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:53:01.274971 3066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:01.275173 3066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:01.275211 3066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:01.276350 3066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:01.276391 3066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:01.276409 3066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:01.276424 3066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:01.276439 3066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:01.277904 3066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:34239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:39107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:01.278138 3066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:01.278343 3066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:01.282958 3071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:01.282975 3074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:01.282965 3072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:01.283226 3066 server_base.cc:1034] running on GCE node
I20260926 22:53:01.283463 3066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:01.283744 3066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:01.284895 3066 hybrid_clock.cc:648] HybridClock initialized: now 1790463181284879 us; error 30 us; skew 500 ppm
Sep 26 22:53:01 dist-test-slave-98hb krb5kdc[3053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463181, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:01.292651 3066 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:53:01.293097 3066 webserver.cc:460] Webserver started at http://127.13.148.62:32847/ using document root <none> and password file <none>
I20260926 22:53:01.293282 3066 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:01.293320 3066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:01.293409 3066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:01.294462 3066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "5df0d15b92a84d338d107318d71f82c6"
format_stamp: "Formatted at 2026-09-26 22:53:01 on dist-test-slave-98hb"
server_key: "356c8170d703d976e642c66b25f99674"
server_key_iv: "1520c6d72cfc957837e276b0c1117559"
server_key_version: "encryptionkey@0"
I20260926 22:53:01.294757 3066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "5df0d15b92a84d338d107318d71f82c6"
format_stamp: "Formatted at 2026-09-26 22:53:01 on dist-test-slave-98hb"
server_key: "356c8170d703d976e642c66b25f99674"
server_key_iv: "1520c6d72cfc957837e276b0c1117559"
server_key_version: "encryptionkey@0"
I20260926 22:53:01.295915 3066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:53:01.296583 3081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:01.296751 3066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:01.296832 3066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "5df0d15b92a84d338d107318d71f82c6"
format_stamp: "Formatted at 2026-09-26 22:53:01 on dist-test-slave-98hb"
server_key: "356c8170d703d976e642c66b25f99674"
server_key_iv: "1520c6d72cfc957837e276b0c1117559"
server_key_version: "encryptionkey@0"
I20260926 22:53:01.296892 3066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:01.320645 3066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:01.322425 3066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:01.322561 3066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:01.326267 3066 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:39107
I20260926 22:53:01.326308 3133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:39107 every 8 connection(s)
I20260926 22:53:01.326629 3066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:53:01.327420 3134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:01.329561 3134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6: Bootstrap starting.
I20260926 22:53:01.330113 3134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:01.330354 3134 log.cc:824] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:01.330951 3134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6: No bootstrap required, opened a new log
I20260926 22:53:01.332471 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 3066
I20260926 22:53:01.332592 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:53:01.332517 3134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0d15b92a84d338d107318d71f82c6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39107 } }
I20260926 22:53:01.332688 3134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:53:01.332728 3134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df0d15b92a84d338d107318d71f82c6, State: Initialized, Role: FOLLOWER
I20260926 22:53:01.332798 13904 external_mini_cluster.cc:1428] Setting key 1f46ab5afd29f35ccc68ec410fd3bc5e
I20260926 22:53:01.332917 3134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0d15b92a84d338d107318d71f82c6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39107 } }
I20260926 22:53:01.332998 3134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:53:01.333021 3134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:53:01.333067 3134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:01.334024 3134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0d15b92a84d338d107318d71f82c6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39107 } }
I20260926 22:53:01.334111 3134 leader_election.cc:302] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5df0d15b92a84d338d107318d71f82c6; no voters:
I20260926 22:53:01.334255 3134 leader_election.cc:288] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:53:01.334306 3139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:53:01.334435 3139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [term 1 LEADER]: Becoming Leader. State: Replica: 5df0d15b92a84d338d107318d71f82c6, State: Running, Role: LEADER
I20260926 22:53:01.334494 3134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:01.334558 3139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0d15b92a84d338d107318d71f82c6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39107 } }
I20260926 22:53:01.334858 3139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5df0d15b92a84d338d107318d71f82c6. Latest consensus state: current_term: 1 leader_uuid: "5df0d15b92a84d338d107318d71f82c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0d15b92a84d338d107318d71f82c6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39107 } } }
I20260926 22:53:01.334920 3139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:01.334887 3140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5df0d15b92a84d338d107318d71f82c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df0d15b92a84d338d107318d71f82c6" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 39107 } } }
I20260926 22:53:01.335052 3140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:01.335611 3144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:53:01.336098 3144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:53:01.338173 3144 catalog_manager.cc:1347] Generated new cluster ID: 36860142f2014e28b379c1e4dcf16aea
I20260926 22:53:01.338231 3144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:53:01.338371 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:01.333254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46053 (local address 127.13.148.62:39107)
0926 22:53:01.333401 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:01.333409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:01.333440 (+ 31us) server_negotiation.cc:407] Connection header received
0926 22:53:01.333540 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:01.333547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:01.333587 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:01.333689 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:01.333941 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:01.335202 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:01.336246 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:01.336846 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:01.336946 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:01.337607 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:01.337610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:01.337611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:53:01.337824 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:01.337864 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:53:01.337878 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:01.337959 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:01.338072 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 22:53:01.338101 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260926 22:53:01.339309 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:53:01.342065 3144 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:53:01.342758 3144 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:53:01.343798 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:01.340638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47027 (local address 127.13.148.62:39107)
0926 22:53:01.340670 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:01.340672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:01.340787 (+ 115us) server_negotiation.cc:407] Connection header received
0926 22:53:01.340846 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:01.340847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:01.340857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:01.340911 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:01.341038 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:01.341754 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:01.342658 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:01.342898 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:01.342950 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:01.343582 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:01.343583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:01.343584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:53:01.343588 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:01.343601 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:53:01.343606 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:01.343653 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:01.343719 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 22:53:01.343730 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:53:01.358641 3144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5df0d15b92a84d338d107318d71f82c6: Generated new TSK 0
I20260926 22:53:01.358855 3144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:53:01.362586 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 3066
W20260926 22:53:01.366443 3051 connection.cc:535] client connection to 127.13.148.62:39107 recv error: Network error: failed to read from TLS socket (remote: 127.13.148.62:39107): Connection reset by peer (error 104)
2026-09-26T22:53:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:53:01 dist-test-slave-98hb krb5kdc[3165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:53:01 dist-test-slave-98hb krb5kdc[3165](info): set up 2 sockets
Sep 26 22:53:01 dist-test-slave-98hb krb5kdc[3165](info): commencing operation
krb5kdc: starting...
W20260926 22:53:02.297941 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:53:02 dist-test-slave-98hb krb5kdc[3165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:53:02.403856 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35007
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:35007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:53:02.482110 3178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:02.482308 3178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:02.482331 3178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:02.483448 3178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:02.483489 3178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:02.483501 3178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:02.483511 3178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:02.483523 3178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:02.484949 3178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:35007
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:02.485162 3178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:02.485363 3178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:53:02.489768 3178 server_base.cc:1034] running on GCE node
W20260926 22:53:02.489828 3184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:02.489791 3186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:02.489909 3183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:02.490149 3178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:02.490429 3178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:02.491565 3178 hybrid_clock.cc:648] HybridClock initialized: now 1790463182491552 us; error 28 us; skew 500 ppm
Sep 26 22:53:02 dist-test-slave-98hb krb5kdc[3165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463182, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:02.493793 3178 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:53:02.494191 3178 webserver.cc:460] Webserver started at http://127.13.148.62:46011/ using document root <none> and password file <none>
I20260926 22:53:02.494382 3178 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:02.494427 3178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:02.494546 3178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:02.495530 3178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3"
format_stamp: "Formatted at 2026-09-26 22:53:02 on dist-test-slave-98hb"
server_key: "358424cbb74f3b756d2c0cc135c2b72f"
server_key_iv: "f86e9a46215c21a170f25482c20376b1"
server_key_version: "encryptionkey@0"
I20260926 22:53:02.495850 3178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3"
format_stamp: "Formatted at 2026-09-26 22:53:02 on dist-test-slave-98hb"
server_key: "358424cbb74f3b756d2c0cc135c2b72f"
server_key_iv: "f86e9a46215c21a170f25482c20376b1"
server_key_version: "encryptionkey@0"
I20260926 22:53:02.497051 3178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:53:02.497699 3193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:02.497844 3178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:02.497924 3178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3"
format_stamp: "Formatted at 2026-09-26 22:53:02 on dist-test-slave-98hb"
server_key: "358424cbb74f3b756d2c0cc135c2b72f"
server_key_iv: "f86e9a46215c21a170f25482c20376b1"
server_key_version: "encryptionkey@0"
I20260926 22:53:02.497987 3178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:02.501004 3178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:02.502707 3178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:02.502832 3178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:02.506407 3178 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:35007
I20260926 22:53:02.506434 3245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:35007 every 8 connection(s)
I20260926 22:53:02.506709 3178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:53:02.507544 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 3178
I20260926 22:53:02.507611 3246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:02.507656 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:53:02.507831 13904 external_mini_cluster.cc:1428] Setting key 1fae0ee19d65115f470626eb1fe89d05
I20260926 22:53:02.510062 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:02.508282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48581 (local address 127.13.148.62:35007)
0926 22:53:02.508447 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:02.508456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:02.508486 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:53:02.508981 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:02.508990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:02.509023 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:02.509170 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:02.509666 (+ 496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:02.509669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:02.509669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:53:02.509674 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:02.509713 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:53:02.509726 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:02.509820 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:02.509854 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 22:53:02.509892 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260926 22:53:02.510715 3246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3: Bootstrap starting.
I20260926 22:53:02.510921 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:53:02.511276 3246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:02.511518 3246 log.cc:824] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:02.512212 3246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3: No bootstrap required, opened a new log
I20260926 22:53:02.513106 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:02.512237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33323 (local address 127.13.148.62:35007)
0926 22:53:02.512274 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:02.512275 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:02.512346 (+ 71us) server_negotiation.cc:407] Connection header received
0926 22:53:02.512415 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:02.512416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:02.512419 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:02.512462 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:02.512907 (+ 445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:02.512908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:02.512909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:53:02.512912 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:02.512922 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:53:02.512926 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:02.512963 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:02.513014 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 22:53:02.513023 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 22:53:02.514376 3246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35007 } }
I20260926 22:53:02.514504 3246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:53:02.514537 3246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2e2cd56bd14d8b90bab77d7fa7fdc3, State: Initialized, Role: FOLLOWER
I20260926 22:53:02.514650 3246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35007 } }
I20260926 22:53:02.514719 3246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:53:02.514748 3246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:53:02.514794 3246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:02.515286 3246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35007 } }
I20260926 22:53:02.515378 3246 leader_election.cc:302] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b2e2cd56bd14d8b90bab77d7fa7fdc3; no voters:
I20260926 22:53:02.515522 3246 leader_election.cc:288] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:53:02.515574 3256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:53:02.515725 3256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7b2e2cd56bd14d8b90bab77d7fa7fdc3, State: Running, Role: LEADER
I20260926 22:53:02.515743 3246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:02.515810 3256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35007 } }
I20260926 22:53:02.516147 3257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35007 } } }
I20260926 22:53:02.516163 3258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b2e2cd56bd14d8b90bab77d7fa7fdc3. Latest consensus state: current_term: 1 leader_uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b2e2cd56bd14d8b90bab77d7fa7fdc3" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 35007 } } }
I20260926 22:53:02.516399 3257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:02.516426 3258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:02.516734 3265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:53:02.517168 3265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:53:02.518520 3265 catalog_manager.cc:1347] Generated new cluster ID: e7bea264ee204ea69d3ec0dec14afbf1
I20260926 22:53:02.518579 3265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:53:02.530798 3265 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:53:02.530952 3265 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:53:02.540444 3265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b2e2cd56bd14d8b90bab77d7fa7fdc3: Generated new TSK 0
I20260926 22:53:02.540622 3265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:53:02.551214 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 3178
2026-09-26T22:53:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:53:02 dist-test-slave-98hb krb5kdc[3277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:53:02 dist-test-slave-98hb krb5kdc[3277](info): set up 2 sockets
Sep 26 22:53:02 dist-test-slave-98hb krb5kdc[3277](info): commencing operation
krb5kdc: starting...
W20260926 22:53:03.493826 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:53:03 dist-test-slave-98hb krb5kdc[3277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:53:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:53:03.602236 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34377
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:53:03.692282 3290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:03.692493 3290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:03.692533 3290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:03.693678 3290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:03.693719 3290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:03.693758 3290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:03.693775 3290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:03.693799 3290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:03.695266 3290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:03.695519 3290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:03.695720 3290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:03.700176 3296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:03.700273 3298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:03.700218 3295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:03.700362 3290 server_base.cc:1034] running on GCE node
I20260926 22:53:03.700596 3290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:03.700845 3290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:03.701972 3290 hybrid_clock.cc:648] HybridClock initialized: now 1790463183701953 us; error 35 us; skew 500 ppm
Sep 26 22:53:03 dist-test-slave-98hb krb5kdc[3277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463183, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:03.704262 3290 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:53:03.704624 3290 webserver.cc:460] Webserver started at http://127.13.148.62:46799/ using document root <none> and password file <none>
I20260926 22:53:03.704810 3290 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:03.704849 3290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:03.704934 3290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:03.705960 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "fc618fd9c67d4ee391a8bf48b37457e9"
format_stamp: "Formatted at 2026-09-26 22:53:03 on dist-test-slave-98hb"
server_key: "79336266960b825ae537611aad715513"
server_key_iv: "69106bb3f8c0e87b981a530395d8d42b"
server_key_version: "encryptionkey@0"
I20260926 22:53:03.706240 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "fc618fd9c67d4ee391a8bf48b37457e9"
format_stamp: "Formatted at 2026-09-26 22:53:03 on dist-test-slave-98hb"
server_key: "79336266960b825ae537611aad715513"
server_key_iv: "69106bb3f8c0e87b981a530395d8d42b"
server_key_version: "encryptionkey@0"
I20260926 22:53:03.707356 3290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:53:03.708000 3305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:03.708168 3290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:03.708247 3290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "fc618fd9c67d4ee391a8bf48b37457e9"
format_stamp: "Formatted at 2026-09-26 22:53:03 on dist-test-slave-98hb"
server_key: "79336266960b825ae537611aad715513"
server_key_iv: "69106bb3f8c0e87b981a530395d8d42b"
server_key_version: "encryptionkey@0"
I20260926 22:53:03.708305 3290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:03.720178 3290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:03.721815 3290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:03.721920 3290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:03.725303 3290 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:34377
I20260926 22:53:03.725356 3357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:34377 every 8 connection(s)
I20260926 22:53:03.725621 3290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:53:03.726362 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 3290
I20260926 22:53:03.726459 3358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:03.726476 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:53:03.726666 13904 external_mini_cluster.cc:1428] Setting key 5319484cbc21a870cf1d4b30875b7f39
I20260926 22:53:03.729342 3358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9: Bootstrap starting.
I20260926 22:53:03.729998 3358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:03.730280 3358 log.cc:824] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:03.730908 3358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9: No bootstrap required, opened a new log
Sep 26 22:53:03 dist-test-slave-98hb krb5kdc[3277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:53:03.733167 3358 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc618fd9c67d4ee391a8bf48b37457e9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34377 } }
I20260926 22:53:03.733321 3358 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:53:03.733345 3358 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc618fd9c67d4ee391a8bf48b37457e9, State: Initialized, Role: FOLLOWER
I20260926 22:53:03.733464 3358 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc618fd9c67d4ee391a8bf48b37457e9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34377 } }
I20260926 22:53:03.733533 3358 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:53:03.733570 3358 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:53:03.733616 3358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:03.734239 3358 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc618fd9c67d4ee391a8bf48b37457e9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34377 } }
I20260926 22:53:03.734350 3358 leader_election.cc:302] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc618fd9c67d4ee391a8bf48b37457e9; no voters:
I20260926 22:53:03.734510 3358 leader_election.cc:288] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:53:03.734551 3363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:53:03.734697 3363 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [term 1 LEADER]: Becoming Leader. State: Replica: fc618fd9c67d4ee391a8bf48b37457e9, State: Running, Role: LEADER
I20260926 22:53:03.734776 3358 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:03.734825 3363 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc618fd9c67d4ee391a8bf48b37457e9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34377 } }
I20260926 22:53:03.735199 3363 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc618fd9c67d4ee391a8bf48b37457e9. Latest consensus state: current_term: 1 leader_uuid: "fc618fd9c67d4ee391a8bf48b37457e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc618fd9c67d4ee391a8bf48b37457e9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34377 } } }
I20260926 22:53:03.735431 3363 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:03.735358 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc618fd9c67d4ee391a8bf48b37457e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc618fd9c67d4ee391a8bf48b37457e9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34377 } } }
I20260926 22:53:03.735787 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:03.735989 3372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:53:03.736626 3372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:53:03.736953 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:03.727126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33713 (local address 127.13.148.62:34377)
0926 22:53:03.727277 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:03.727284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:03.727307 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:53:03.727831 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:03.727837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:03.727884 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:03.727987 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:03.728691 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:03.729582 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:03.730630 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:03.731053 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:03.731131 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:03.733518 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:03.733522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:03.733522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:03.733532 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:03.735285 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:03.735648 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:03.735651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:03.735651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:03.735766 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:03.736060 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:03.736061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:03.736061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:03.736346 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:03.736493 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:03.736596 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 22:53:03.736812 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260926 22:53:03.737882 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:53:03.742159 3372 catalog_manager.cc:1347] Generated new cluster ID: ae44e3efe21b4728a2ca6f0b485be5aa
I20260926 22:53:03.742206 3372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:53:03.744674 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:03.739087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42591 (local address 127.13.148.62:34377)
0926 22:53:03.739121 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:03.739123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:03.739180 (+ 57us) server_negotiation.cc:407] Connection header received
0926 22:53:03.739245 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:03.739246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:03.739254 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:03.739295 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:03.739855 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:03.740294 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:03.741280 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:03.741467 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:03.741511 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:03.742590 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:03.742591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:03.742591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:03.742598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:03.743736 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:03.744026 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:03.744027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:03.744027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:03.744076 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:03.744231 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:03.744232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:03.744232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:03.744337 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:03.744413 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:03.744515 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 22:53:03.744582 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:53:03.752694 3372 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:53:03.753191 3372 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:53:03.763085 3372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc618fd9c67d4ee391a8bf48b37457e9: Generated new TSK 0
I20260926 22:53:03.763262 3372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:53:03.766351 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 3290
2026-09-26T22:53:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:53:03 dist-test-slave-98hb krb5kdc[3389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:53:03 dist-test-slave-98hb krb5kdc[3389](info): set up 2 sockets
Sep 26 22:53:03 dist-test-slave-98hb krb5kdc[3389](info): commencing operation
krb5kdc: starting...
W20260926 22:53:04.721892 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:53:04 dist-test-slave-98hb krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:53:04.827848 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34871
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:53:04.904618 3402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:04.904817 3402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:04.904846 3402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:04.906000 3402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:04.906040 3402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:04.906059 3402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:04.906073 3402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:04.906087 3402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:04.907533 3402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:34871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:04.907773 3402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:04.907971 3402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:04.912434 3407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:04.912495 3402 server_base.cc:1034] running on GCE node
W20260926 22:53:04.912472 3410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:04.912456 3408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:04.912859 3402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:04.913128 3402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:04.914261 3402 hybrid_clock.cc:648] HybridClock initialized: now 1790463184914236 us; error 33 us; skew 500 ppm
Sep 26 22:53:04 dist-test-slave-98hb krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463184, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:04.916440 3402 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:53:04.916846 3402 webserver.cc:460] Webserver started at http://127.13.148.62:42607/ using document root <none> and password file <none>
I20260926 22:53:04.917038 3402 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:04.917080 3402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:04.917191 3402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:04.918222 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "7b63f15240fa46fe979867e8f888ecb9"
format_stamp: "Formatted at 2026-09-26 22:53:04 on dist-test-slave-98hb"
server_key: "8a7f6ae472d393867a005c9cc9079dab"
server_key_iv: "3bc108fabbb4aea2d4989e01f2264919"
server_key_version: "encryptionkey@0"
I20260926 22:53:04.918538 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "7b63f15240fa46fe979867e8f888ecb9"
format_stamp: "Formatted at 2026-09-26 22:53:04 on dist-test-slave-98hb"
server_key: "8a7f6ae472d393867a005c9cc9079dab"
server_key_iv: "3bc108fabbb4aea2d4989e01f2264919"
server_key_version: "encryptionkey@0"
I20260926 22:53:04.919713 3402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:53:04.920331 3417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:04.920495 3402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:04.920580 3402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "7b63f15240fa46fe979867e8f888ecb9"
format_stamp: "Formatted at 2026-09-26 22:53:04 on dist-test-slave-98hb"
server_key: "8a7f6ae472d393867a005c9cc9079dab"
server_key_iv: "3bc108fabbb4aea2d4989e01f2264919"
server_key_version: "encryptionkey@0"
I20260926 22:53:04.920630 3402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:04.930369 3402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:04.932243 3402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:04.932345 3402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:04.935853 3402 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:34871
I20260926 22:53:04.935900 3469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:34871 every 8 connection(s)
I20260926 22:53:04.936204 3402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:53:04.937053 3470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:04.939247 3470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9: Bootstrap starting.
I20260926 22:53:04.939793 3470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:04.940054 3470 log.cc:824] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:04.940647 3470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9: No bootstrap required, opened a new log
I20260926 22:53:04.941551 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 3402
I20260926 22:53:04.941673 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:53:04.941891 13904 external_mini_cluster.cc:1428] Setting key a05540ce58f9b9ac502a76b6e32db781
I20260926 22:53:04.942570 3470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b63f15240fa46fe979867e8f888ecb9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34871 } }
I20260926 22:53:04.942706 3470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:53:04.942736 3470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b63f15240fa46fe979867e8f888ecb9, State: Initialized, Role: FOLLOWER
I20260926 22:53:04.942866 3470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b63f15240fa46fe979867e8f888ecb9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34871 } }
I20260926 22:53:04.942948 3470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:53:04.942976 3470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:53:04.943059 3470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:04.943583 3470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b63f15240fa46fe979867e8f888ecb9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34871 } }
I20260926 22:53:04.943675 3470 leader_election.cc:302] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b63f15240fa46fe979867e8f888ecb9; no voters:
I20260926 22:53:04.943830 3470 leader_election.cc:288] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:53:04.943861 3475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:53:04.943961 3475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [term 1 LEADER]: Becoming Leader. State: Replica: 7b63f15240fa46fe979867e8f888ecb9, State: Running, Role: LEADER
I20260926 22:53:04.944078 3470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:04.944096 3475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b63f15240fa46fe979867e8f888ecb9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34871 } }
I20260926 22:53:04.944471 3476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b63f15240fa46fe979867e8f888ecb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b63f15240fa46fe979867e8f888ecb9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34871 } } }
I20260926 22:53:04.944584 3476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:04.944474 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b63f15240fa46fe979867e8f888ecb9. Latest consensus state: current_term: 1 leader_uuid: "7b63f15240fa46fe979867e8f888ecb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b63f15240fa46fe979867e8f888ecb9" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 34871 } } }
I20260926 22:53:04.944864 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:04.945082 3483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:53:04.945612 3483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:53:04.947131 3483 catalog_manager.cc:1347] Generated new cluster ID: 5391563303fa4066a1e05729c1c020e1
I20260926 22:53:04.947197 3483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:53:04.947316 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:04.942381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35841 (local address 127.13.148.62:34871)
0926 22:53:04.942540 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:04.942549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:04.942577 (+ 28us) server_negotiation.cc:407] Connection header received
0926 22:53:04.942666 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:04.942671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:04.942710 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:04.942835 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:04.943168 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:04.944268 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:04.945328 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:04.945925 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:04.946027 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:04.946624 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:04.946628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:04.946628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:53:04.946817 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:04.946850 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:53:04.946861 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:04.946914 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:04.946989 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 22:53:04.947050 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260926 22:53:04.948091 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:53:04.952142 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:04.949262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46495 (local address 127.13.148.62:34871)
0926 22:53:04.949294 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:04.949296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:04.949886 (+ 590us) server_negotiation.cc:407] Connection header received
0926 22:53:04.949939 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:04.949940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:04.949950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:04.949993 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:04.950098 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:04.950529 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:04.951177 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:04.951364 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:04.951414 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:04.951932 (+ 518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:04.951933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:04.951933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:53:04.951937 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:04.951951 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:53:04.951954 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:04.951996 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:04.952047 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 22:53:04.952074 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:53:04.952392 3483 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:53:04.953076 3483 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:53:04.957787 3483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b63f15240fa46fe979867e8f888ecb9: Generated new TSK 0
I20260926 22:53:04.957933 3483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:53:04.961760 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 3402
2026-09-26T22:53:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:53:05 dist-test-slave-98hb krb5kdc[3502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:53:05 dist-test-slave-98hb krb5kdc[3502](info): set up 2 sockets
Sep 26 22:53:05 dist-test-slave-98hb krb5kdc[3502](info): commencing operation
krb5kdc: starting...
W20260926 22:53:05.921926 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:53:05 dist-test-slave-98hb krb5kdc[3502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:53:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:53:06.030417 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36009
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:44305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:53:06.108228 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:06.108438 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:06.108469 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:06.109616 3515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:06.109663 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:06.109680 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:06.109696 3515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:06.109710 3515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:06.111182 3515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:44305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:06.111425 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:06.111629 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:06.116063 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:06.116113 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:06.116125 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:06.116267 3515 server_base.cc:1034] running on GCE node
I20260926 22:53:06.116492 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:06.116730 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:06.117864 3515 hybrid_clock.cc:648] HybridClock initialized: now 1790463186117855 us; error 25 us; skew 500 ppm
Sep 26 22:53:06 dist-test-slave-98hb krb5kdc[3502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463186, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:06.120198 3515 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:53:06.120589 3515 webserver.cc:460] Webserver started at http://127.13.148.62:36869/ using document root <none> and password file <none>
I20260926 22:53:06.120795 3515 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:06.120838 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:06.120944 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:06.121980 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "5e148ff562924c1daff2b47c6e4fcb62"
format_stamp: "Formatted at 2026-09-26 22:53:06 on dist-test-slave-98hb"
server_key: "7030fc64d8957d3b207e38f7c9b9c62a"
server_key_iv: "6638c4ab9a3cdb352e20bc9e6c111999"
server_key_version: "encryptionkey@0"
I20260926 22:53:06.122290 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "5e148ff562924c1daff2b47c6e4fcb62"
format_stamp: "Formatted at 2026-09-26 22:53:06 on dist-test-slave-98hb"
server_key: "7030fc64d8957d3b207e38f7c9b9c62a"
server_key_iv: "6638c4ab9a3cdb352e20bc9e6c111999"
server_key_version: "encryptionkey@0"
I20260926 22:53:06.123457 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 22:53:06.124123 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:06.124269 3515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:06.124337 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "5e148ff562924c1daff2b47c6e4fcb62"
format_stamp: "Formatted at 2026-09-26 22:53:06 on dist-test-slave-98hb"
server_key: "7030fc64d8957d3b207e38f7c9b9c62a"
server_key_iv: "6638c4ab9a3cdb352e20bc9e6c111999"
server_key_version: "encryptionkey@0"
I20260926 22:53:06.124401 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:06.127511 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:06.129135 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:06.129246 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:06.132722 3515 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36009
I20260926 22:53:06.132753 3582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36009 every 8 connection(s)
I20260926 22:53:06.133028 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:53:06.133940 3583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:06.134163 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 3515
I20260926 22:53:06.134275 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:53:06.134464 13904 external_mini_cluster.cc:1428] Setting key 5a1ad64ef2bf57110a5412dde393ec00
I20260926 22:53:06.136607 3583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62: Bootstrap starting.
I20260926 22:53:06.136656 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:06.134935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60299 (local address 127.13.148.62:36009)
0926 22:53:06.135100 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:06.135108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:06.135131 (+ 23us) server_negotiation.cc:407] Connection header received
0926 22:53:06.135531 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:06.135537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:06.135561 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:06.135674 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:06.136260 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:06.136263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:06.136263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:53:06.136266 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:06.136305 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:53:06.136315 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:06.136366 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:06.136448 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 22:53:06.136479 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260926 22:53:06.137229 3583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:06.137490 3583 log.cc:824] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:06.137538 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:53:06.138190 3583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62: No bootstrap required, opened a new log
I20260926 22:53:06.139815 3586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:06.139005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53173 (local address 127.13.148.62:36009)
0926 22:53:06.139040 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:06.139042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:06.139093 (+ 51us) server_negotiation.cc:407] Connection header received
0926 22:53:06.139160 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:06.139161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:06.139164 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:06.139204 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:06.139645 (+ 441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:06.139645 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:06.139646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 22:53:06.139648 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:06.139659 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 22:53:06.139662 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:06.139700 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:06.139753 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 22:53:06.139763 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:53:06.140375 3583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e148ff562924c1daff2b47c6e4fcb62" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36009 } }
I20260926 22:53:06.140501 3583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 22:53:06.140520 3583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e148ff562924c1daff2b47c6e4fcb62, State: Initialized, Role: FOLLOWER
I20260926 22:53:06.140609 3583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e148ff562924c1daff2b47c6e4fcb62" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36009 } }
I20260926 22:53:06.140668 3583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 22:53:06.140686 3583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 22:53:06.140715 3583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:06.141157 3583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e148ff562924c1daff2b47c6e4fcb62" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36009 } }
I20260926 22:53:06.141237 3583 leader_election.cc:302] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e148ff562924c1daff2b47c6e4fcb62; no voters:
I20260926 22:53:06.141345 3583 leader_election.cc:288] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 22:53:06.141402 3593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:53:06.141539 3583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:06.141571 3593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [term 1 LEADER]: Becoming Leader. State: Replica: 5e148ff562924c1daff2b47c6e4fcb62, State: Running, Role: LEADER
I20260926 22:53:06.141705 3593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e148ff562924c1daff2b47c6e4fcb62" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36009 } }
I20260926 22:53:06.142068 3595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e148ff562924c1daff2b47c6e4fcb62. Latest consensus state: current_term: 1 leader_uuid: "5e148ff562924c1daff2b47c6e4fcb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e148ff562924c1daff2b47c6e4fcb62" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36009 } } }
I20260926 22:53:06.142081 3594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e148ff562924c1daff2b47c6e4fcb62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e148ff562924c1daff2b47c6e4fcb62" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36009 } } }
I20260926 22:53:06.142167 3595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:06.142175 3594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62 [sys.catalog]: This master's current role is: LEADER
W20260926 22:53:06.143271 3609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:53:06.143327 3609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:53:06.143410 3602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:53:06.143580 3602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:53:06.144893 3602 catalog_manager.cc:1347] Generated new cluster ID: 79f27cf136864f52a4feefddc329cdaa
I20260926 22:53:06.144934 3602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:53:06.164989 3602 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:53:06.165179 3602 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:53:06.169194 3602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e148ff562924c1daff2b47c6e4fcb62: Generated new TSK 0
I20260926 22:53:06.169376 3602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:53:06.171726 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 3515
2026-09-26T22:53:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1210 ms)
[----------] 18 tests from AuthTokenIssuingTest (21750 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:53:06 dist-test-slave-98hb krb5kdc[3614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:53:06 dist-test-slave-98hb krb5kdc[3614](info): set up 2 sockets
Sep 26 22:53:06 dist-test-slave-98hb krb5kdc[3614](info): commencing operation
krb5kdc: starting...
W20260926 22:53:07.121964 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:53:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:53:07.232451 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46727
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46727,127.13.148.61:44431,127.13.148.60:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:53:07.309571 3627 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:53:07.309808 3627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:07.309834 3627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:07.309859 3627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:07.310962 3627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:07.311000 3627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:07.311018 3627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:07.311033 3627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:07.311048 3627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:07.312485 3627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46727,127.13.148.61:44431,127.13.148.60:39895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:46727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:07.312731 3627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:07.312937 3627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 22:53:07.317396 3627 server_base.cc:1034] running on GCE node
W20260926 22:53:07.317404 3632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:07.317436 3633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:07.317517 3635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:07.317862 3627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:07.318125 3627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:07.319263 3627 hybrid_clock.cc:648] HybridClock initialized: now 1790463187319251 us; error 25 us; skew 500 ppm
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:07.327036 3627 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:53:07.327401 3627 webserver.cc:460] Webserver started at http://127.13.148.62:33893/ using document root <none> and password file <none>
I20260926 22:53:07.327582 3627 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:07.327622 3627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:07.327710 3627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:07.328670 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "2cfda1560f674ab59ab8bb4794d7ccc5"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "2520d127dea918ddc1e3ffdee96a021e"
server_key_iv: "72b8e884ac6f0af5a05ab992afdec126"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.328953 3627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "2cfda1560f674ab59ab8bb4794d7ccc5"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "2520d127dea918ddc1e3ffdee96a021e"
server_key_iv: "72b8e884ac6f0af5a05ab992afdec126"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.330116 3627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:53:07.330735 3642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:07.330884 3627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:07.330963 3627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "2cfda1560f674ab59ab8bb4794d7ccc5"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "2520d127dea918ddc1e3ffdee96a021e"
server_key_iv: "72b8e884ac6f0af5a05ab992afdec126"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.331020 3627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:07.342000 3627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:07.343883 3627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:07.344007 3627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:07.347584 3627 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:46727
I20260926 22:53:07.347606 3694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:46727 every 8 connection(s)
I20260926 22:53:07.347905 3627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:53:07.348771 3695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:07.350239 3695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:53:07.356451 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 3627
I20260926 22:53:07.356577 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:53:07.356756 13904 external_mini_cluster.cc:1428] Setting key 0f0afb0df48332f7ebc9d5f4c3402834
I20260926 22:53:07.358553 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.350814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46727 (local address 127.0.0.1:49824)
0926 22:53:07.350951 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.350983 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.351420 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.351671 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.351674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.351919 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.352057 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.352062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.353028 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.353029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.353666 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.353667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.354132 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.354133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.354195 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.354557 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.354564 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.355778 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.357276 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.357282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.357284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.357456 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.357649 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.357649 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.357650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.357721 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.358045 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.358049 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.358106 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.358183 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.358397 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":91,"thread_start_us":68,"threads_started":1}
I20260926 22:53:07.358557 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.350960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.13.148.62:46727)
0926 22:53:07.351063 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.351066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.351096 (+ 30us) server_negotiation.cc:407] Connection header received
0926 22:53:07.351564 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.351566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.351588 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.351660 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.352149 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.352943 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.353755 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.354046 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.354137 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.355855 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.355862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.355862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.355868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.357205 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.357522 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.357525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.357525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.357594 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.357797 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.357798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.357798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.357985 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.358054 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.358221 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.358327 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":38,"threads_started":1}
I20260926 22:53:07.359159 3695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:07.359469 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.359370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53546)
0926 22:53:07.359408 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.359448 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 22:53:07.359575 3645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
I20260926 22:53:07.359964 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.359881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53560)
0926 22:53:07.359918 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.359940 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 22:53:07.360055 3695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:44431: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.13.148.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
I20260926 22:53:07.377767 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:44431
--webserver_interface=127.13.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46727,127.13.148.61:44431,127.13.148.60:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:53:07.409251 3695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } attempt: 1
I20260926 22:53:07.409757 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.409596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53572)
0926 22:53:07.409665 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.409704 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:53:07.410080 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.410009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53582)
0926 22:53:07.410042 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.410065 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 22:53:07.410192 3695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:44431: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
W20260926 22:53:07.457366 3702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:53:07.457589 3702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:07.457612 3702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:07.457647 3702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:07.458882 3702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:07.458925 3702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:07.458943 3702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:07.458958 3702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:07.458974 3702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:07.460440 3702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46727,127.13.148.61:44431,127.13.148.60:39895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:07.460675 3702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:07.460881 3702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:07.465390 3707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:07.465391 3710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:07.465402 3708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:07.465535 3702 server_base.cc:1034] running on GCE node
I20260926 22:53:07.465843 3702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:07.466128 3702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:07.467260 3702 hybrid_clock.cc:648] HybridClock initialized: now 1790463187467251 us; error 26 us; skew 500 ppm
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:07.469484 3702 init.cc:377] Logged in from keytab as kudu/127.13.148.61@KRBTEST.COM (short username kudu)
I20260926 22:53:07.469942 3702 webserver.cc:460] Webserver started at http://127.13.148.61:37531/ using document root <none> and password file <none>
I20260926 22:53:07.470153 3702 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:07.470196 3702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:07.470319 3702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:07.471292 3702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/data/instance:
uuid: "be72224b5e5f4e5c81afe9b0d4da19ce"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "f46a5b40e261cd41758e7b269ad516d4"
server_key_iv: "a1eb46c49d82913934da18dd6fdd375c"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.471608 3702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/wal/instance:
uuid: "be72224b5e5f4e5c81afe9b0d4da19ce"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "f46a5b40e261cd41758e7b269ad516d4"
server_key_iv: "a1eb46c49d82913934da18dd6fdd375c"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.472769 3702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:53:07.473392 3717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:07.473529 3702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:07.473594 3702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/wal
uuid: "be72224b5e5f4e5c81afe9b0d4da19ce"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "f46a5b40e261cd41758e7b269ad516d4"
server_key_iv: "a1eb46c49d82913934da18dd6fdd375c"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.473659 3702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:07.478310 3695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } attempt: 2
I20260926 22:53:07.478637 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.478526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53594)
0926 22:53:07.478575 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.478607 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:53:07.478971 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.478857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53606)
0926 22:53:07.478905 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.478940 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 22:53:07.479063 3695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:44431: Network error: Client connection negotiation failed: client connection to 127.13.148.61:44431: connect: Connection refused (error 111)
I20260926 22:53:07.498041 3702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:07.499647 3702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:07.499758 3702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:07.503484 3702 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:44431
I20260926 22:53:07.503516 3769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:44431 every 8 connection(s)
I20260926 22:53:07.503842 3702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
I20260926 22:53:07.504719 3770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:07.506251 3770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:07.506631 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 3702
I20260926 22:53:07.506747 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-1/wal/instance
I20260926 22:53:07.506924 13904 external_mini_cluster.cc:1428] Setting key de40716ac84be76b5fa4510cb0ff3cfe
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.148.60@KRBTEST.COM; defaulting to no policy
I20260926 22:53:07.515012 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.506945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.13.148.62:46727)
0926 22:53:07.506996 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.506999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.507201 (+ 202us) server_negotiation.cc:407] Connection header received
0926 22:53:07.507644 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.507645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.507657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.507723 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.508281 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.508745 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.510056 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.510255 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.510329 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.512450 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.512452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.512452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.512459 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.513641 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.514160 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.514161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.514162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.514208 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.514427 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.514428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.514428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.514533 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.514587 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.514841 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.514921 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:53:07.515174 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.506926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46727 (local address 127.0.0.1:49840)
0926 22:53:07.507079 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.507119 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.507562 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.507776 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.507779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.508004 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.508184 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.508188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.508827 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.508828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.509976 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.509979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.510355 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.510356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.510509 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.510948 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.510957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.512366 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.513715 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.513722 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.513724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.513958 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.514270 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.514271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.514271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.514374 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.514587 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.514594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.514727 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.514813 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.514973 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260926 22:53:07.515623 3770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.60@KRBTEST.COM" created.
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:53:07.522251 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.515875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53614)
0926 22:53:07.515918 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.515927 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.515965 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.516167 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.516168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.516291 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.516331 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.516332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.517189 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.517189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.517884 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.517886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.518465 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.518465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.518510 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.518827 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.518831 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.519886 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.521352 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.521352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.521353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.521495 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.521659 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.521660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.521660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.521711 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.521967 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.521968 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.522002 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.522060 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.522150 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 22:53:07.522315 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.515923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.13.148.61:44431)
0926 22:53:07.516058 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.516062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.516075 (+ 13us) server_negotiation.cc:407] Connection header received
0926 22:53:07.516090 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.516091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.516101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.516167 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.516405 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.517114 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.517955 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.518399 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.518472 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.519954 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.519962 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.519963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.519968 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.521281 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.521555 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.521558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.521558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.521603 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.521786 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.521787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.521787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.521910 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.521977 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.522092 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.522201 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260926 22:53:07.522949 3770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:07.523257 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.523147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:32958)
0926 22:53:07.523191 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.523235 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 22:53:07.523371 3720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
I20260926 22:53:07.523773 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.523691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:32972)
0926 22:53:07.523726 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.523748 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 22:53:07.523864 3770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:39895: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
I20260926 22:53:07.530213 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:39895
--webserver_interface=127.13.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:46727,127.13.148.61:44431,127.13.148.60:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:53:07.570188 3695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } attempt: 3
I20260926 22:53:07.573060 3770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } attempt: 1
I20260926 22:53:07.574033 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.573385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:32984)
0926 22:53:07.573959 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.573996 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:53:07.574393 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.574287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:32988)
0926 22:53:07.574335 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.574368 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 22:53:07.574522 3770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:39895: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:53:07.577242 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.570593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.13.148.61:44431)
0926 22:53:07.570655 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.570657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.570667 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:53:07.570734 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.570735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.570745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.570791 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.571136 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.571620 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.572623 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.572834 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.572879 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.574945 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.574946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.574946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.574952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.576202 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.576547 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.576548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.576548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.576594 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.576804 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.576804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.576805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.576927 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.576980 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.577127 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.577190 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:53:07.577291 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.570523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53626)
0926 22:53:07.570589 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.570604 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.570671 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.570810 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.570811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.570997 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.571075 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.571078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.571713 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.571715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.572548 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.572553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.572892 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.572893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.572967 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.573470 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.573474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.574869 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.576271 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.576272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.576273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.576483 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.576641 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.576642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.576642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.576721 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.576978 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.576979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.577031 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.577099 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.577194 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 22:53:07.577629 3695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:07.577975 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.577877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:32996)
0926 22:53:07.577918 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.577951 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 22:53:07.578248 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.578171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:32998)
0926 22:53:07.578208 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.578231 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 22:53:07.578331 3695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:39895: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
I20260926 22:53:07.609463 3695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } attempt: 1
W20260926 22:53:07.609571 3777 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:53:07.609846 3777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:07.609882 3777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:07.609907 3777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 22:53:07.609949 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.609822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:33010)
0926 22:53:07.609875 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.609917 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 22:53:07.610275 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.610196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:33016)
0926 22:53:07.610234 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.610258 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 22:53:07.610360 3695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:39895: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
W20260926 22:53:07.611191 3777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:07.611233 3777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:07.611245 3777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:07.611256 3777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:07.611267 3777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:07.612641 3777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:46727,127.13.148.61:44431,127.13.148.60:39895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:07.612813 3777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:07.612972 3777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:07.617331 3783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:07.617399 3777 server_base.cc:1034] running on GCE node
W20260926 22:53:07.617466 3782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:07.617343 3785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:07.617785 3777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:07.618065 3777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:07.619191 3777 hybrid_clock.cc:648] HybridClock initialized: now 1790463187619183 us; error 25 us; skew 500 ppm
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:07.621428 3777 init.cc:377] Logged in from keytab as kudu/127.13.148.60@KRBTEST.COM (short username kudu)
I20260926 22:53:07.621881 3777 webserver.cc:460] Webserver started at http://127.13.148.60:41569/ using document root <none> and password file <none>
I20260926 22:53:07.622073 3777 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:07.622114 3777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:07.622246 3777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:07.623250 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/data/instance:
uuid: "8070fe5b040a49c38f16e3d7f4263622"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "4d5d610c3f9cb2948a8f0914faf48fb1"
server_key_iv: "e5b66455bff07dac100e44979a9a0832"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.623574 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/wal/instance:
uuid: "8070fe5b040a49c38f16e3d7f4263622"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "4d5d610c3f9cb2948a8f0914faf48fb1"
server_key_iv: "e5b66455bff07dac100e44979a9a0832"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.624713 3777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 22:53:07.625327 3792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:07.625463 3777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 22:53:07.625541 3777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/wal
uuid: "8070fe5b040a49c38f16e3d7f4263622"
format_stamp: "Formatted at 2026-09-26 22:53:07 on dist-test-slave-98hb"
server_key: "4d5d610c3f9cb2948a8f0914faf48fb1"
server_key_iv: "e5b66455bff07dac100e44979a9a0832"
server_key_version: "encryptionkey@0"
I20260926 22:53:07.625605 3777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:07.641582 3777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:07.642655 3770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } attempt: 2
I20260926 22:53:07.643179 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.643066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:33032)
0926 22:53:07.643121 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.643152 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 22:53:07.643390 3777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:07.643513 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.643422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:33034)
0926 22:53:07.643468 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.643492 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 22:53:07.643522 3777 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260926 22:53:07.643607 3770 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:39895: Network error: Client connection negotiation failed: client connection to 127.13.148.60:39895: connect: Connection refused (error 111)
I20260926 22:53:07.647164 3777 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:39895
I20260926 22:53:07.647202 3844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:39895 every 8 connection(s)
I20260926 22:53:07.647504 3777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
I20260926 22:53:07.648378 3845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:07.649825 3845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:07.654232 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 3777
I20260926 22:53:07.654345 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790462935413609-13904-0/minicluster-data/master-2/wal/instance
I20260926 22:53:07.654521 13904 external_mini_cluster.cc:1428] Setting key 67774b2615b698bea0a5233ed0dea59b
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:53:07.657893 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.650443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.13.148.62:46727)
0926 22:53:07.650492 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.650494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.650684 (+ 190us) server_negotiation.cc:407] Connection header received
0926 22:53:07.651147 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.651148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.651160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.651210 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.651780 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.652242 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.653162 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.653340 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.653393 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.655425 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.655427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.655427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.655434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.656714 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.657064 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.657065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.657065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.657131 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.657318 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.657318 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.657319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.657438 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.657496 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.657757 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.657832 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 22:53:07.658079 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.650408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:46727 (local address 127.0.0.1:49848)
0926 22:53:07.650570 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.650610 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.651053 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.651288 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.651291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.651511 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.651699 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.651703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.652316 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.652317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.653085 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.653087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.653414 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.653415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.653527 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.653951 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.653959 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.655349 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.656785 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.656790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.656792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.657002 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.657187 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.657188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.657188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.657272 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.657487 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.657490 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.657608 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.657709 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.657870 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260926 22:53:07.658674 3845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:53:07.663030 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.654863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.13.148.60:39895)
0926 22:53:07.655008 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.655013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.655058 (+ 45us) server_negotiation.cc:407] Connection header received
0926 22:53:07.655144 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.655149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.655171 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.655228 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.655915 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.656709 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.657578 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.658011 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.658080 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.660493 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.660499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.660499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.660505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.661907 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.662257 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.662259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.662260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.662313 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.662498 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.662498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.662499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.662660 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.662762 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.662868 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.662937 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260926 22:53:07.665091 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.658886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:44431 (local address 127.0.0.1:53628)
0926 22:53:07.658930 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.658941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.658984 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.659079 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.659079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.659222 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.659271 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.659272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.660028 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.660029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.660666 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.660668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.661073 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.661074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.661127 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.661464 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.661468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.662576 (+ 1108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.664223 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.664224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.664224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.664368 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.664535 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.664536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.664536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.664602 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.664868 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.664869 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.664900 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.664949 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.665013 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 22:53:07.665149 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.658948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.13.148.61:44431)
0926 22:53:07.658987 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.658988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.659006 (+ 18us) server_negotiation.cc:407] Connection header received
0926 22:53:07.659028 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.659029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.659037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.659072 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.659330 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.659955 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.660744 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.661000 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.661062 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.662673 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.662674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.662674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.662680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.664158 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.664436 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.664436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.664437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.664485 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.664656 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.664656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.664657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.664784 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.664841 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.664998 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.665077 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:53:07.665576 3845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:53:07.670354 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.663751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.13.148.62:46727)
0926 22:53:07.663782 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.663784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.663792 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:53:07.663850 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.663851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.663858 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.663888 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.664388 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.664992 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.665794 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.666005 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.666053 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.667904 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.667905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.667905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.667912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.669343 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.669614 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.669615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.669616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.669669 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.669876 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.669877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.669877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.669979 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.670105 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.670193 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.670266 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 22:53:07.671715 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.665801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:33054)
0926 22:53:07.665834 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.665843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.665872 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.666264 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.666264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.666383 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.666411 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.666412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.666957 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.666958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.667489 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.667492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.667824 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.667825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.667870 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.668198 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.668201 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.669436 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.670900 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.670901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.670901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.671104 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.671261 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.671261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.671262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.671319 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.671496 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.671496 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.671535 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.671598 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.671661 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 22:53:07.671809 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.666103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.13.148.60:39895)
0926 22:53:07.666155 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.666157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.666163 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:53:07.666179 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.666180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.666189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.666235 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.666466 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.666894 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.667559 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.667769 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.667819 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.669502 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.669503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.669503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.669509 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.670830 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.671162 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.671162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.671162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.671206 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.671373 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.671373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.671373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.671449 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.671488 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.671638 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.671727 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":29}
I20260926 22:53:07.673427 3845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622: Bootstrap starting.
I20260926 22:53:07.674453 3845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:07.674706 3845 log.cc:824] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622: Log is configured to *not* fsync() on all Append() calls
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:53:07.675349 3845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622: No bootstrap required, opened a new log
I20260926 22:53:07.676743 3845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } }
I20260926 22:53:07.676875 3845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8070fe5b040a49c38f16e3d7f4263622, State: Initialized, Role: FOLLOWER
I20260926 22:53:07.676959 3845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } }
I20260926 22:53:07.677268 3845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:07.677559 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.670884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.13.148.61:44431)
0926 22:53:07.670919 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.670921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.671004 (+ 83us) server_negotiation.cc:407] Connection header received
0926 22:53:07.671033 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.671034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.671042 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.671073 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.671545 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.672114 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.672831 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.673007 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.673048 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.675167 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.675168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.675168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.675175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.676479 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.676776 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.676777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.676777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.676828 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.676997 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.676998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.676999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.677153 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.677309 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.677403 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.677474 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:53:07.677229 3851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.677654 3851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:07.678905 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 22:53:07.678987 3862 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:53:07.679028 3862 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:53:07.679260 3777 master_runner.cc:186] Error getting master registration for 127.13.148.62:46727: OK, instance_id { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" instance_seqno: 1790463187346610 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:53:07.681416 3777 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:53:07.681875 3669 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856
I20260926 22:53:07.685467 3695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } attempt: 2
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:53:07.691484 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.685709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:33060)
0926 22:53:07.685804 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.685816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.685860 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.685974 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.685974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.686112 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.686158 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.686159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.686743 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.686743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.687251 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.687252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.687558 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.687559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.687609 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.687965 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.687970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.689171 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.690548 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.690549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.690550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.690764 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.690915 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.690915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.690916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.690973 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.691179 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.691180 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.691240 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.691302 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.691388 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 22:53:07.691490 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.685799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.13.148.60:39895)
0926 22:53:07.685852 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.685855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.685864 (+ 9us) server_negotiation.cc:407] Connection header received
0926 22:53:07.685918 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.685919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.685927 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.685971 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.686217 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.686681 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.687320 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.687506 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.687555 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.689246 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.689248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.689248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.689254 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.690492 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.690825 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.690825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.690826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.690869 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.691019 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.691020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.691020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.691126 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.691172 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.691335 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.691405 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 22:53:07.692945 3695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5: Bootstrap starting.
I20260926 22:53:07.693568 3695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:07.693878 3695 log.cc:824] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:07.694664 3695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5: No bootstrap required, opened a new log
I20260926 22:53:07.695927 3695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } }
I20260926 22:53:07.696067 3695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfda1560f674ab59ab8bb4794d7ccc5, State: Initialized, Role: FOLLOWER
I20260926 22:53:07.696149 3695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } }
I20260926 22:53:07.696391 3865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.696437 3695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:07.696475 3865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:07.696483 3669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 22:53:07.696633 3669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:07.697326 3669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } }
I20260926 22:53:07.697919 3819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8070fe5b040a49c38f16e3d7f4263622"
I20260926 22:53:07.698040 3819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [term 0 FOLLOWER]: Advancing to term 1
W20260926 22:53:07.698734 3876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:53:07.698799 3876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:53:07.698863 3819 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cfda1560f674ab59ab8bb4794d7ccc5 in term 1.
I20260926 22:53:07.699051 3645 leader_election.cc:302] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfda1560f674ab59ab8bb4794d7ccc5, 8070fe5b040a49c38f16e3d7f4263622; no voters:
I20260926 22:53:07.699158 3669 leader_election.cc:288] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [CANDIDATE]: Term 1 election: Requested vote from peers be72224b5e5f4e5c81afe9b0d4da19ce (127.13.148.61:44431), 8070fe5b040a49c38f16e3d7f4263622 (127.13.148.60:39895)
I20260926 22:53:07.699177 3865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:53:07.699330 3865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [term 1 LEADER]: Becoming Leader. State: Replica: 2cfda1560f674ab59ab8bb4794d7ccc5, State: Running, Role: LEADER
I20260926 22:53:07.699402 3865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } }
I20260926 22:53:07.700783 3878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cfda1560f674ab59ab8bb4794d7ccc5. Latest consensus state: current_term: 1 leader_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.700866 3878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:07.698726 3627 master_runner.cc:186] Error getting master registration for 127.13.148.61:44431: OK, instance_id { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" instance_seqno: 1790463187502426 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:53:07.701292 3627 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:53:07.708026 3885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:53:07.708256 3885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:53:07.711094 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.701046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.13.148.61:44431)
0926 22:53:07.701087 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.701089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.701914 (+ 825us) server_negotiation.cc:407] Connection header received
0926 22:53:07.702023 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.702023 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.702035 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.702082 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.703075 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.703772 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.705163 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.705448 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.705489 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.707578 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.707579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.707579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.707586 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.709503 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.710071 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.710072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.710073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.710150 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.710436 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.710437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.710437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.710571 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.710651 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.710926 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.711004 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 22:53:07.712445 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.700738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.13.148.62:46727)
0926 22:53:07.700775 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.700777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.701319 (+ 542us) server_negotiation.cc:407] Connection header received
0926 22:53:07.701408 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.701409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.701419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.701445 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.702207 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.702933 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.703938 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.704220 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.704257 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.705708 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.705709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.705710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.705716 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.707282 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.707681 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.707682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.707683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.707737 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.711505 (+ 3768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.711506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.711506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.711640 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.711704 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.712285 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.712359 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 22:53:07.713024 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.701239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.13.148.60:39895)
0926 22:53:07.701270 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.701272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.702084 (+ 812us) server_negotiation.cc:407] Connection header received
0926 22:53:07.702173 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.702174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.702183 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.702211 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.705809 (+ 3598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.706452 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.707388 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.707659 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.707697 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.709193 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.709194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.709194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.709200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.710735 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.711207 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.711208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.711208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.711257 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.711914 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.711915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.711915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.712034 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.712093 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.712867 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.712939 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:53:07.714546 3819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [term 1 FOLLOWER]: Refusing update from remote peer 2cfda1560f674ab59ab8bb4794d7ccc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:53:07.714859 3878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:53:07.716218 3851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cfda1560f674ab59ab8bb4794d7ccc5. Latest consensus state: current_term: 1 leader_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.716300 3851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:07.722748 3865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.722846 3865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:07.722970 3851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.723049 3851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:07.723028 3888 mvcc.cc:205] Tried to move back new op lower bound from 7333737216876580864 to 7333737216817303552. Current Snapshot: MvccSnapshot[applied={T|T < 7333737216876580864}]
I20260926 22:53:07.723241 3889 mvcc.cc:205] Tried to move back new op lower bound from 7333737216876580864 to 7333737216817303552. Current Snapshot: MvccSnapshot[applied={T|T < 7333737216876580864}]
I20260926 22:53:07.733403 3885 catalog_manager.cc:1347] Generated new cluster ID: aaeab9f2f2fe402ab814a7e3173a1081
I20260926 22:53:07.733464 3885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:53:07.734738 3770 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } attempt: 3
Sep 26 22:53:07 dist-test-slave-98hb krb5kdc[3614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463187, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:53:07.750669 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.734963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:39895 (local address 127.0.0.1:53022)
0926 22:53:07.735038 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:07.735048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:07.735083 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:07.735418 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:07.735419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:07.735595 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:07.735642 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.735644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.736589 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.736590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.737395 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:07.737397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.737972 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.737973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:07.738029 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.738476 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:07.738480 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:07.739984 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:07.741761 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.741762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.741763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.741964 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.749944 (+ 7980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:07.749945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:07.749945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:07.750023 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:07.750348 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:07.750349 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:07.750410 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 22:53:07.750467 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 22:53:07.750565 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 22:53:07.751122 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:07.735182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.13.148.60:39895)
0926 22:53:07.735277 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.735278 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.735286 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:53:07.735320 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.735321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.735330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.735375 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.735733 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.736503 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.737498 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:07.737865 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:07.737922 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:07.740086 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:07.740087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:07.740088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:07.740095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:07.741671 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.749788 (+ 8117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.749789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.749790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.749861 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:07.750102 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:07.750104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:07.750104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:07.750248 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:07.750307 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:07.750954 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 22:53:07.751035 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 22:53:07.752904 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce: Bootstrap starting.
I20260926 22:53:07.753648 3770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:07.754757 3770 log.cc:824] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:07.755307 3885 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:53:07.755898 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce: No bootstrap required, opened a new log
I20260926 22:53:07.755976 3885 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:53:07.757941 3770 raft_consensus.cc:348] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } }
I20260926 22:53:07.758072 3770 raft_consensus.cc:729] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be72224b5e5f4e5c81afe9b0d4da19ce, State: Initialized, Role: FOLLOWER
I20260926 22:53:07.758155 3770 consensus_queue.cc:260] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } }
I20260926 22:53:07.758457 3770 sys_catalog.cc:564] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:07.759960 3743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:07.759681 3744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce"
I20260926 22:53:07.760808 3743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [term 1 FOLLOWER]: Refusing update from remote peer 2cfda1560f674ab59ab8bb4794d7ccc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:53:07.761541 3744 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cfda1560f674ab59ab8bb4794d7ccc5 in term 1.
I20260926 22:53:07.762195 3892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:53:07.764488 3895 sys_catalog.cc:455] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.764582 3895 sys_catalog.cc:458] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:07.764647 3895 sys_catalog.cc:455] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.764693 3895 sys_catalog.cc:458] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:07.764771 3910 sys_catalog.cc:455] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cfda1560f674ab59ab8bb4794d7ccc5. Latest consensus state: current_term: 1 leader_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.764822 3910 sys_catalog.cc:458] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:07.765507 3906 catalog_manager.cc:1259] Loaded cluster ID: aaeab9f2f2fe402ab814a7e3173a1081
I20260926 22:53:07.765556 3906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce: loading cluster ID for follower catalog manager: success
I20260926 22:53:07.766690 3906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce: acquiring CA information for follower catalog manager: success
W20260926 22:53:07.766808 3906 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260926 22:53:07.769807 3885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5: Generated new TSK 0
I20260926 22:53:07.769979 3885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 22:53:07.770469 3878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfda1560f674ab59ab8bb4794d7ccc5" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 46727 } } peers { permanent_uuid: "be72224b5e5f4e5c81afe9b0d4da19ce" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 44431 } } peers { permanent_uuid: "8070fe5b040a49c38f16e3d7f4263622" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 39895 } } }
I20260926 22:53:07.770560 3878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfda1560f674ab59ab8bb4794d7ccc5 [sys.catalog]: This master's current role is: LEADER
W20260926 22:53:07.793298 3698 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.791756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.13.148.62:46727)
0926 22:53:07.791797 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.791799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.792665 (+ 866us) server_negotiation.cc:407] Connection header received
0926 22:53:07.792770 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.792771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.792782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.792810 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.793230 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49366: BlockingRecv error: recv got EOF from 127.0.0.1:49366 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 22:53:07.794351 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.792091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.13.148.61:44431)
0926 22:53:07.792130 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.792132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.793500 (+ 1368us) server_negotiation.cc:407] Connection header received
0926 22:53:07.793814 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.793815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.793826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.793854 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.794292 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42130: BlockingRecv error: recv got EOF from 127.0.0.1:42130 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 22:53:07.794787 3849 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.792388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.13.148.60:39895)
0926 22:53:07.792425 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.792427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.793647 (+ 1220us) server_negotiation.cc:407] Connection header received
0926 22:53:07.793915 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.793916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.793926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.793953 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.794731 (+ 778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53032: BlockingRecv error: recv got EOF from 127.0.0.1:53032 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 22:53:07.797073 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.796319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.13.148.61:44431)
0926 22:53:07.796353 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.796355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.796576 (+ 221us) server_negotiation.cc:407] Connection header received
0926 22:53:07.796690 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.796691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.796708 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.796736 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.797026 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42140: BlockingRecv error: recv got EOF from 127.0.0.1:42140 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 22:53:07.799957 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.799228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.13.148.61:44431)
0926 22:53:07.799258 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.799259 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.799490 (+ 231us) server_negotiation.cc:407] Connection header received
0926 22:53:07.799579 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.799579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.799588 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.799624 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.799918 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42154: BlockingRecv error: recv got EOF from 127.0.0.1:42154 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 22:53:07.803578 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.802914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.13.148.61:44431)
0926 22:53:07.802944 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.802945 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.803157 (+ 212us) server_negotiation.cc:407] Connection header received
0926 22:53:07.803241 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.803242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.803251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.803276 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.803540 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42164: BlockingRecv error: recv got EOF from 127.0.0.1:42164 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 22:53:07.809252 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.808552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.13.148.61:44431)
0926 22:53:07.808585 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.808587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.808815 (+ 228us) server_negotiation.cc:407] Connection header received
0926 22:53:07.808897 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.808898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.808907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.808933 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.809212 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42180: BlockingRecv error: recv got EOF from 127.0.0.1:42180 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:53:07.819175 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.818368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.13.148.61:44431)
0926 22:53:07.818405 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.818407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.818672 (+ 265us) server_negotiation.cc:407] Connection header received
0926 22:53:07.818761 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.818762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.818772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.818798 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.819124 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42196: BlockingRecv error: recv got EOF from 127.0.0.1:42196 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 22:53:07.837101 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.836286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.13.148.61:44431)
0926 22:53:07.836324 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.836326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.836560 (+ 234us) server_negotiation.cc:407] Connection header received
0926 22:53:07.836661 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.836662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.836672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.836699 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.837044 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42204: BlockingRecv error: recv got EOF from 127.0.0.1:42204 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 22:53:07.870870 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.870213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42218 (local address 127.13.148.61:44431)
0926 22:53:07.870255 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.870257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.870392 (+ 135us) server_negotiation.cc:407] Connection header received
0926 22:53:07.870446 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.870447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.870459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.870505 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.870810 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42218: BlockingRecv error: recv got EOF from 127.0.0.1:42218 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 22:53:07.936439 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:07.935857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.13.148.61:44431)
0926 22:53:07.935940 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:07.935941 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:07.935958 (+ 17us) server_negotiation.cc:407] Connection header received
0926 22:53:07.936024 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:07.936025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:07.936037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:07.936102 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:07.936390 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42232: BlockingRecv error: recv got EOF from 127.0.0.1:42232 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 22:53:08.066377 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:08.065673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.13.148.61:44431)
0926 22:53:08.065764 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:08.065766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:08.065858 (+ 92us) server_negotiation.cc:407] Connection header received
0926 22:53:08.065934 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:08.065935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:08.065946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:08.065990 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:08.066314 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42240: BlockingRecv error: recv got EOF from 127.0.0.1:42240 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 22:53:08.323951 3774 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:08.323408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.13.148.61:44431)
0926 22:53:08.323477 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:08.323479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:08.323491 (+ 12us) server_negotiation.cc:407] Connection header received
0926 22:53:08.323557 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:08.323558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:08.323568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:08.323607 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:08.323889 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42250: BlockingRecv error: recv got EOF from 127.0.0.1:42250 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:53:08.679435 3862 catalog_manager.cc:1259] Loaded cluster ID: aaeab9f2f2fe402ab814a7e3173a1081
I20260926 22:53:08.679507 3862 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622: loading cluster ID for follower catalog manager: success
I20260926 22:53:08.680266 3862 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622: acquiring CA information for follower catalog manager: success
I20260926 22:53:08.680513 3862 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8070fe5b040a49c38f16e3d7f4263622: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:53:08.767373 3906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be72224b5e5f4e5c81afe9b0d4da19ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:53:08.839324 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:08.836982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.13.148.61:44431)
0926 22:53:08.837124 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:08.837126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:08.837137 (+ 11us) server_negotiation.cc:407] Connection header received
0926 22:53:08.837200 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:08.837201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:08.837251 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:08.837316 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:08.837437 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.837953 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.838665 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.838866 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.838922 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:08.839031 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:08.839140 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:08.839196 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:08.839211 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 22:53:08.839230 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260926 22:53:08.843081 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:08.840916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53038 (local address 127.13.148.60:39895)
0926 22:53:08.841021 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:08.841023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:08.841063 (+ 40us) server_negotiation.cc:407] Connection header received
0926 22:53:08.841127 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:08.841127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:08.841165 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:08.841230 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:08.841300 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.841822 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.842467 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.842678 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.842727 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:08.842834 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:08.842934 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:08.842985 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:08.843003 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 22:53:08.843020 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
I20260926 22:53:08.847260 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:08.844630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.13.148.61:44431)
0926 22:53:08.844679 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:08.844681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:08.844852 (+ 171us) server_negotiation.cc:407] Connection header received
0926 22:53:08.844929 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:08.844930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:08.844940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:08.844984 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:08.845076 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.845622 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.846657 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.846868 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.846927 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:08.847029 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:08.847058 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:08.847099 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:08.847160 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 22:53:08.847185 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:53:08.847510 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:08.844792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53044 (local address 127.13.148.60:39895)
0926 22:53:08.844825 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:08.844827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:08.844869 (+ 42us) server_negotiation.cc:407] Connection header received
0926 22:53:08.844964 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:08.844965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:08.844975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:08.845018 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:08.845179 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.845891 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.846686 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.846969 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.847029 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:08.847258 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:08.847288 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:08.847334 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:08.847427 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 22:53:08.847450 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 22:53:08.847916 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:08.844514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.13.148.62:46727)
0926 22:53:08.844697 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:08.844699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:08.844721 (+ 22us) server_negotiation.cc:407] Connection header received
0926 22:53:08.844767 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:08.844767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:08.844819 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:08.845025 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:08.845065 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.845852 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.846886 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:08.847159 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:08.847314 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:08.847487 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:08.847586 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:08.847796 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:08.847814 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 22:53:08.847841 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":49,"threads_started":1}
I20260926 22:53:08.849329 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 3627
I20260926 22:53:08.853972 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 3702
I20260926 22:53:08.857936 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 3777
2026-09-26T22:53:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2686 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 22:53:08 dist-test-slave-98hb krb5kdc[3995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 22:53:08 dist-test-slave-98hb krb5kdc[3995](info): set up 2 sockets
Sep 26 22:53:08 dist-test-slave-98hb krb5kdc[3995](info): commencing operation
krb5kdc: starting...
W20260926 22:53:09.806082 13904 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/test-user.keytab.
Sep 26 22:53:09 dist-test-slave-98hb krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T22:53:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T22:53:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.148.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
Entry for principal kudu/127.13.148.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab.
I20260926 22:53:09.913128 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36709
--webserver_interface=127.13.148.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36709,127.13.148.61:42183,127.13.148.60:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 22:53:09.990015 4008 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:53:09.990204 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:09.990219 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:09.990237 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:09.991341 4008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:09.991380 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:09.991393 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:09.991405 4008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:09.991415 4008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:09.992861 4008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36709,127.13.148.61:42183,127.13.148.60:43929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.62.keytab
--principal=kudu/127.13.148.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.62:36709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.148.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:09.993077 4008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:09.993286 4008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:09.997583 4016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:09.997596 4013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:09.997620 4014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:09.997838 4008 server_base.cc:1034] running on GCE node
I20260926 22:53:09.998051 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:09.998327 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:09.999457 4008 hybrid_clock.cc:648] HybridClock initialized: now 1790463189999444 us; error 30 us; skew 500 ppm
Sep 26 22:53:09 dist-test-slave-98hb krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463189, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:10.001667 4008 init.cc:377] Logged in from keytab as kudu/127.13.148.62@KRBTEST.COM (short username kudu)
I20260926 22:53:10.002086 4008 webserver.cc:460] Webserver started at http://127.13.148.62:36201/ using document root <none> and password file <none>
I20260926 22:53:10.002274 4008 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:10.002311 4008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:10.002398 4008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:10.003350 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/data/instance:
uuid: "d640c55f27c44c77b55d705ec7361d96"
format_stamp: "Formatted at 2026-09-26 22:53:09 on dist-test-slave-98hb"
server_key: "a633b094cbfe08f62244c6be70dfb439"
server_key_iv: "3899e36998f631bdb412f88f6cbd636a"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.003628 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/wal/instance:
uuid: "d640c55f27c44c77b55d705ec7361d96"
format_stamp: "Formatted at 2026-09-26 22:53:09 on dist-test-slave-98hb"
server_key: "a633b094cbfe08f62244c6be70dfb439"
server_key_iv: "3899e36998f631bdb412f88f6cbd636a"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.004742 4008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 22:53:10.005343 4023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:10.005481 4008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:10.005559 4008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/wal
uuid: "d640c55f27c44c77b55d705ec7361d96"
format_stamp: "Formatted at 2026-09-26 22:53:09 on dist-test-slave-98hb"
server_key: "a633b094cbfe08f62244c6be70dfb439"
server_key_iv: "3899e36998f631bdb412f88f6cbd636a"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.005611 4008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:10.017046 4008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:10.018651 4008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:10.018752 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:10.022248 4008 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.62:36709
I20260926 22:53:10.022284 4075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.62:36709 every 8 connection(s)
I20260926 22:53:10.022574 4008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/data/info.pb
I20260926 22:53:10.023470 4076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:10.024866 4076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:10.027109 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 4008
I20260926 22:53:10.027228 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-0/wal/instance
I20260926 22:53:10.027410 13904 external_mini_cluster.cc:1428] Setting key 8c199abee1d422dc086eec945af59e13
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463189, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:53:10.034214 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.025466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36709 (local address 127.0.0.1:60092)
0926 22:53:10.025628 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.025667 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.026127 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.026351 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.026353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.026570 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.026736 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.026740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.028039 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.028040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.028701 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.028703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.029146 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.029147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.029210 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.029667 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.029677 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.031093 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.032867 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.032872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.032874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.033049 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.033294 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.033295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.033295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.033363 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.033780 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.033784 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.033836 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.033903 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.034019 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260926 22:53:10.034219 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.025495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.13.148.62:36709)
0926 22:53:10.025698 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.025701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.025723 (+ 22us) server_negotiation.cc:407] Connection header received
0926 22:53:10.026245 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.026247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.026264 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.026336 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.026801 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.027958 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.028778 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.029062 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.029170 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.031179 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.031186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.031186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.031194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.032792 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.033122 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.033125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.033125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.033235 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.033415 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.033416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.033417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.033682 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.033790 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.033957 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.034090 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":181,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.13.148.61@KRBTEST.COM; defaulting to no policy
I20260926 22:53:10.035015 4076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:10.035307 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.035209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33218)
0926 22:53:10.035248 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.035287 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 22:53:10.035435 4027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
I20260926 22:53:10.035778 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.035699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33232)
0926 22:53:10.035735 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.035764 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 22:53:10.035861 4076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:42183: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
Entry for principal kudu/127.13.148.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab.
I20260926 22:53:10.052327 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:42183
--webserver_interface=127.13.148.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36709,127.13.148.61:42183,127.13.148.60:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:53:10.085084 4076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } attempt: 1
I20260926 22:53:10.085541 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.085429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33242)
0926 22:53:10.085479 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.085509 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 22:53:10.085861 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.085792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33258)
0926 22:53:10.085818 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.085842 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260926 22:53:10.085944 4076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:42183: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
W20260926 22:53:10.130646 4083 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:53:10.130861 4083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:10.130885 4083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:10.130909 4083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:10.132032 4083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:10.132071 4083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:10.132088 4083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:10.132103 4083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:10.132118 4083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:10.133576 4083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36709,127.13.148.61:42183,127.13.148.60:43929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.61.keytab
--principal=kudu/127.13.148.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.61:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.148.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:10.133841 4083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:10.134048 4083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:10.138612 4091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:10.138636 4089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:10.138721 4083 server_base.cc:1034] running on GCE node
W20260926 22:53:10.138703 4088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:10.138991 4083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:10.139271 4083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:10.140407 4083 hybrid_clock.cc:648] HybridClock initialized: now 1790463190140398 us; error 38 us; skew 500 ppm
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463190, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:10.142699 4083 init.cc:377] Logged in from keytab as kudu/127.13.148.61@KRBTEST.COM (short username kudu)
I20260926 22:53:10.143112 4083 webserver.cc:460] Webserver started at http://127.13.148.61:45687/ using document root <none> and password file <none>
I20260926 22:53:10.143316 4083 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:10.143362 4083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:10.143481 4083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:10.144474 4083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/data/instance:
uuid: "bd76a0298d5a47c0821c507f92524e70"
format_stamp: "Formatted at 2026-09-26 22:53:10 on dist-test-slave-98hb"
server_key: "f59f15d7db4190956f86abdf8d081c62"
server_key_iv: "8b1c4e6dc2223ab869da3f454671887b"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.144791 4083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/wal/instance:
uuid: "bd76a0298d5a47c0821c507f92524e70"
format_stamp: "Formatted at 2026-09-26 22:53:10 on dist-test-slave-98hb"
server_key: "f59f15d7db4190956f86abdf8d081c62"
server_key_iv: "8b1c4e6dc2223ab869da3f454671887b"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.145972 4083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 22:53:10.146601 4098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:10.146744 4083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:10.146821 4083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/wal
uuid: "bd76a0298d5a47c0821c507f92524e70"
format_stamp: "Formatted at 2026-09-26 22:53:10 on dist-test-slave-98hb"
server_key: "f59f15d7db4190956f86abdf8d081c62"
server_key_iv: "8b1c4e6dc2223ab869da3f454671887b"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.146876 4083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:10.154072 4076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } attempt: 2
I20260926 22:53:10.154439 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.154329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33272)
0926 22:53:10.154377 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.154410 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 22:53:10.154772 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.154682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33276)
0926 22:53:10.154722 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.154747 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 22:53:10.154848 4076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.61:42183: Network error: Client connection negotiation failed: client connection to 127.13.148.61:42183: connect: Connection refused (error 111)
I20260926 22:53:10.173199 4083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:10.174923 4083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:10.175053 4083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:10.178817 4083 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.61:42183
I20260926 22:53:10.178853 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.61:42183 every 8 connection(s)
I20260926 22:53:10.179181 4083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/data/info.pb
I20260926 22:53:10.180064 4151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:10.181536 4151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:10.186546 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 4083
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463190, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:53:10.186651 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-1/wal/instance
I20260926 22:53:10.186847 13904 external_mini_cluster.cc:1428] Setting key dfb53ffdf16bbabf45ac81f5a7223648
I20260926 22:53:10.189790 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.182179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.13.148.62:36709)
0926 22:53:10.182228 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.182230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.182432 (+ 202us) server_negotiation.cc:407] Connection header received
0926 22:53:10.182877 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.182878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.182888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.182939 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.183484 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.183951 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.184856 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.185037 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.185081 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.187151 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.187152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.187153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.187160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.188568 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.188938 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.188939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.188939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.189000 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.189193 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.189193 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.189194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.189323 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.189385 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.189590 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.189681 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:53:10.189906 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.182159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36709 (local address 127.0.0.1:60108)
0926 22:53:10.182321 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.182363 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.182799 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.182995 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.182998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.183237 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.183421 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.183425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.184014 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.184014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.184795 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.184797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.185098 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.185098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.185198 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.185589 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.185596 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.187063 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.188642 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.188647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.188649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.188868 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.189057 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.189058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.189058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.189144 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.189377 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.189379 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.189485 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.189561 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.189678 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260926 22:53:10.190431 4151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.148.60@KRBTEST.COM; defaulting to no policy
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463190, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:53:10.197047 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.190679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33284)
0926 22:53:10.190874 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.190884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.190935 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.191059 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.191060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.191194 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.191251 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.191252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.192037 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.192037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.192794 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.192797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.193209 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.193211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.193281 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.193597 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.193602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.194815 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.196216 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.196217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.196217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.196420 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.196587 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.196588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.196588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.196631 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.196837 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.196837 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.196870 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.196916 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.196987 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260926 22:53:10.197144 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.190740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.13.148.61:42183)
0926 22:53:10.190874 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.190877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.190929 (+ 52us) server_negotiation.cc:407] Connection header received
0926 22:53:10.190987 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.190988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.190998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.191054 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.191316 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.191961 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.192868 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.193145 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.193190 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.194906 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.194913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.194914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.194919 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.196151 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.196481 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.196483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.196483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.196533 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.196664 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.196665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.196665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.196789 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.196859 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.196940 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.197049 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.148.60@KRBTEST.COM" created.
I20260926 22:53:10.197677 4151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:10.198000 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.197902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51460)
0926 22:53:10.197940 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.197981 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 22:53:10.198119 4100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
I20260926 22:53:10.198480 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.198406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51466)
0926 22:53:10.198443 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.198466 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 22:53:10.198589 4151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:43929: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
Entry for principal kudu/127.13.148.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab.
I20260926 22:53:10.209707 13904 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskClAdwy/build/release/bin/kudu
/tmp/dist-test-taskClAdwy/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:43929
--webserver_interface=127.13.148.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.148.20:40373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.148.62:36709,127.13.148.61:42183,127.13.148.60:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 22:53:10.245968 4076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } attempt: 3
I20260926 22:53:10.247789 4151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } attempt: 1
I20260926 22:53:10.248154 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.248027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51472)
0926 22:53:10.248078 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.248117 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:53:10.248518 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.248429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51482)
0926 22:53:10.248468 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.248498 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 22:53:10.248594 4151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:43929: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463189, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:53:10.253170 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.246412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.13.148.61:42183)
0926 22:53:10.246489 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.246492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.246505 (+ 13us) server_negotiation.cc:407] Connection header received
0926 22:53:10.246534 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.246535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.246547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.246600 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.246891 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.247657 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.248515 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.248824 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.248877 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.250808 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.250810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.250810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.250817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.252177 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.252476 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.252477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.252478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.252539 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.252720 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.252722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.252722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.252834 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.252892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.253017 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.253085 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 22:53:10.253176 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.246364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33288)
0926 22:53:10.246422 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.246435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.246489 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.246614 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.246615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.246767 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.246828 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.246830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.247742 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.247743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.248424 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.248428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.248898 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.248899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.248996 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.249380 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.249386 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.250724 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.252249 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.252250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.252250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.252410 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.252597 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.252597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.252598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.252660 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.252891 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.252892 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.252935 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.252989 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.253083 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 22:53:10.253530 4076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:10.253861 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.253776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51498)
0926 22:53:10.253808 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.253839 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 22:53:10.254109 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.254053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51500)
0926 22:53:10.254075 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.254094 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 22:53:10.254186 4076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:43929: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
I20260926 22:53:10.285324 4076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } attempt: 1
I20260926 22:53:10.285786 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.285652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51502)
0926 22:53:10.285709 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.285759 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 22:53:10.286115 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.286035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51508)
0926 22:53:10.286073 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.286098 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 22:53:10.286201 4076 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:43929: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
W20260926 22:53:10.289469 4158 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 22:53:10.289660 4158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 22:53:10.289683 4158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 22:53:10.289717 4158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 22:53:10.290850 4158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 22:53:10.290889 4158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 22:53:10.290907 4158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 22:53:10.290925 4158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 22:53:10.290946 4158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 22:53:10.292379 4158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.148.20:40373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.148.62:36709,127.13.148.61:42183,127.13.148.60:43929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/krb5kdc/kudu_127.13.148.60.keytab
--principal=kudu/127.13.148.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.148.60:43929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.148.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 21:54:42 UTC on ccycloud.quasar-awvddz.root.comops.site
build id 4675485
I20260926 22:53:10.292604 4158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 22:53:10.292799 4158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 22:53:10.297197 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 22:53:10.297201 4163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:10.297325 4158 server_base.cc:1034] running on GCE node
W20260926 22:53:10.297339 4164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 22:53:10.297695 4158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 22:53:10.297973 4158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 22:53:10.299093 4158 hybrid_clock.cc:648] HybridClock initialized: now 1790463190299082 us; error 28 us; skew 500 ppm
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463190, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 22:53:10.301285 4158 init.cc:377] Logged in from keytab as kudu/127.13.148.60@KRBTEST.COM (short username kudu)
I20260926 22:53:10.301666 4158 webserver.cc:460] Webserver started at http://127.13.148.60:46267/ using document root <none> and password file <none>
I20260926 22:53:10.301872 4158 fs_manager.cc:359] Metadata directory not provided
I20260926 22:53:10.301913 4158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 22:53:10.302028 4158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 22:53:10.302996 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/data/instance:
uuid: "5e835ad474194259a0af1eab89906dbf"
format_stamp: "Formatted at 2026-09-26 22:53:10 on dist-test-slave-98hb"
server_key: "f46ac3f498df8c478eb29f2f3e31beba"
server_key_iv: "cf2d64d03929f92f55bf4311d8d3a709"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.303318 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/wal/instance:
uuid: "5e835ad474194259a0af1eab89906dbf"
format_stamp: "Formatted at 2026-09-26 22:53:10 on dist-test-slave-98hb"
server_key: "f46ac3f498df8c478eb29f2f3e31beba"
server_key_iv: "cf2d64d03929f92f55bf4311d8d3a709"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.304450 4158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 22:53:10.305073 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 22:53:10.305219 4158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 22:53:10.305299 4158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/data,/tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/wal
uuid: "5e835ad474194259a0af1eab89906dbf"
format_stamp: "Formatted at 2026-09-26 22:53:10 on dist-test-slave-98hb"
server_key: "f46ac3f498df8c478eb29f2f3e31beba"
server_key_iv: "cf2d64d03929f92f55bf4311d8d3a709"
server_key_version: "encryptionkey@0"
I20260926 22:53:10.305365 4158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 22:53:10.314030 4158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 22:53:10.315580 4158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 22:53:10.315696 4158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 22:53:10.316708 4151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } attempt: 2
I20260926 22:53:10.317015 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.316912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51514)
0926 22:53:10.316960 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.316986 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 22:53:10.317359 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.317228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51522)
0926 22:53:10.317289 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.317327 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 22:53:10.317461 4151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.148.60:43929: Network error: Client connection negotiation failed: client connection to 127.13.148.60:43929: connect: Connection refused (error 111)
I20260926 22:53:10.319346 4158 rpc_server.cc:307] RPC server started. Bound to: 127.13.148.60:43929
I20260926 22:53:10.319375 4225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.148.60:43929 every 8 connection(s)
I20260926 22:53:10.319664 4158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/data/info.pb
I20260926 22:53:10.320525 4226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 22:53:10.321964 4226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:10.323602 13904 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskClAdwy/build/release/bin/kudu as pid 4158
I20260926 22:53:10.323699 13904 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskClAdwy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790462935413609-13904-0/minicluster-data/master-2/wal/instance
I20260926 22:53:10.323899 13904 external_mini_cluster.cc:1428] Setting key de40e9deb2f5a66da498b505141b9490
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463190, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:53:10.330147 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.322567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.13.148.62:36709)
0926 22:53:10.322617 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.322619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.322794 (+ 175us) server_negotiation.cc:407] Connection header received
0926 22:53:10.323251 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.323253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.323265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.323321 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.323910 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.324518 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.325428 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.325622 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.325675 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.327736 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.327737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.327737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.327743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.328935 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.329282 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.329284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.329285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.329340 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.329601 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.329602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.329602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.329699 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.329773 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.330027 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.330087 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 22:53:10.330433 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.322533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.62:36709 (local address 127.0.0.1:60118)
0926 22:53:10.322688 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.322721 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.323163 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.323387 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.323390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.323612 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.323834 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.323838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.324597 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.324598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.325351 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.325353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.325706 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.325707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.325843 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.326261 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.326269 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.327660 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.329009 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.329016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.329018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.329231 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.329407 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.329408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.329408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.329534 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.329771 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.329778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.329889 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.329994 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.330165 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260926 22:53:10.330981 4226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:10.331848 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.324223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.13.148.60:43929)
0926 22:53:10.324507 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.324510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.324524 (+ 14us) server_negotiation.cc:407] Connection header received
0926 22:53:10.324539 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.324541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.324555 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.324610 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.325149 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.325876 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.326558 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.326929 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.326999 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.329015 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.329022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.329022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.329029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.330669 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.331015 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.331017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.331017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.331073 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.331323 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.331324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.331324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.331487 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.331617 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.331683 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.331754 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":73,"threads_started":1}
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463190, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.62@KRBTEST.COM
I20260926 22:53:10.336730 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.331192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.61:42183 (local address 127.0.0.1:33296)
0926 22:53:10.331232 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.331240 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.331267 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.331416 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.331416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.331511 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.331537 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.331538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.332198 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.332199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.332830 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.332832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.333166 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.333167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.333208 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.333526 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.333529 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.334555 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.335948 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.335949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.335949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.336099 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.336284 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.336285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.336285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.336324 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.336532 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.336533 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.336567 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.336615 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.336676 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 22:53:10.336792 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.331278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.13.148.61:42183)
0926 22:53:10.331321 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.331322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.331329 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:53:10.331350 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.331351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.331360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.331411 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.331603 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.332126 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.332905 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.333097 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.333154 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.334625 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.334626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.334627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.334633 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.335874 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.336156 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.336157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.336158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.336215 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.336360 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.336361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.336361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.336478 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.336540 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.336641 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.336712 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:53:10.337049 4226 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } has no permanent_uuid. Determining permanent_uuid...
I20260926 22:53:10.339284 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.332508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.13.148.62:36709)
0926 22:53:10.332540 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.332542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.332549 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:53:10.332595 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.332596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.332602 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.332640 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.333100 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.333675 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.334376 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.334560 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.334610 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.336913 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.336915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.336915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.336921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.338382 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.338667 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.338668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.338668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.338719 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.338872 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.338872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.338873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.338968 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.339100 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.339135 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.339201 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463190, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.60@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:53:10.342840 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.337258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51542)
0926 22:53:10.337298 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.337306 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.337338 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.337429 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.337430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.337556 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.337586 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.337587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.338418 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.338418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.338858 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.338859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.339190 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.339191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.339233 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.339537 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.339542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.340564 (+ 1022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.341986 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.341987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.341988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.342178 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.342354 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.342354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.342355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.342404 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.342621 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.342622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.342663 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.342714 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.342786 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 22:53:10.342939 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.337283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.13.148.60:43929)
0926 22:53:10.337343 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.337344 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.337366 (+ 22us) server_negotiation.cc:407] Connection header received
0926 22:53:10.337381 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.337381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.337389 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.337428 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.337669 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.338347 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.338929 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.339136 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.339188 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.340636 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.340637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.340637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.340644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.341926 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.342242 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.342243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.342243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.342295 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.342462 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.342462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.342463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.342561 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.342621 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.342750 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.342836 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":44}
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.148.61@KRBTEST.COM
I20260926 22:53:10.344223 4226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf: Bootstrap starting.
I20260926 22:53:10.344991 4226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:10.345278 4226 log.cc:824] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:10.346004 4226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf: No bootstrap required, opened a new log
I20260926 22:53:10.346714 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.339876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.13.148.61:42183)
0926 22:53:10.339913 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.339915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.339921 (+ 6us) server_negotiation.cc:407] Connection header received
0926 22:53:10.339938 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.339939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.339947 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.339996 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.340430 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.341027 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.341930 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.342192 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.342243 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.344391 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.344392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.344393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.344398 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.345724 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.346027 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.346028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.346028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.346083 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.346258 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.346259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.346259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.346375 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.346524 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.346557 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.346622 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:53:10.347992 13904 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 22:53:10.347906 4226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } }
I20260926 22:53:10.348042 4226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e835ad474194259a0af1eab89906dbf, State: Initialized, Role: FOLLOWER
I20260926 22:53:10.348119 4226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } }
I20260926 22:53:10.348366 4232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.348409 4226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:10.348456 4232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:10.349056 4050 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d640c55f27c44c77b55d705ec7361d96"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132
W20260926 22:53:10.350093 4243 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:53:10.350147 4243 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:53:10.350399 4158 master_runner.cc:186] Error getting master registration for 127.13.148.62:36709: OK, instance_id { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" instance_seqno: 1790463190021238 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:53:10.352957 4158 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 22:53:10.361327 4076 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } attempt: 2
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463189, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.62@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:53:10.368448 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.361582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51546)
0926 22:53:10.361630 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.361641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.361680 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.361839 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.361839 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.362003 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.362049 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.362051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.362901 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.362902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.363682 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.363684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.364111 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.364111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.364160 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.364563 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.364569 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.365838 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.367444 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.367445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.367445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.367642 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.367841 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.367842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.367843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.367898 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.368166 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.368166 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.368206 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.368263 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.368345 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 22:53:10.368458 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.361646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.13.148.60:43929)
0926 22:53:10.361697 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.361699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.361728 (+ 29us) server_negotiation.cc:407] Connection header received
0926 22:53:10.361764 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.361765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.361774 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.361825 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.362110 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.362834 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.363753 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.364049 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.364113 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.365910 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.365911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.365911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.365918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.367372 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.367707 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.367708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.367708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.367763 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.367948 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.367949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.367949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.368104 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.368163 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.368295 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.368368 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 22:53:10.370136 4076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96: Bootstrap starting.
I20260926 22:53:10.370828 4076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:10.371104 4076 log.cc:824] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:10.371872 4076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96: No bootstrap required, opened a new log
I20260926 22:53:10.373133 4076 raft_consensus.cc:348] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } }
I20260926 22:53:10.373271 4076 raft_consensus.cc:729] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d640c55f27c44c77b55d705ec7361d96, State: Initialized, Role: FOLLOWER
I20260926 22:53:10.373348 4076 consensus_queue.cc:260] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } }
I20260926 22:53:10.373651 4076 sys_catalog.cc:564] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:10.373695 4050 raft_consensus.cc:482] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 22:53:10.373762 4050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:10.374507 4050 raft_consensus.cc:504] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } }
I20260926 22:53:10.374747 4050 leader_election.cc:288] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [CANDIDATE]: Term 1 election: Requested vote from peers bd76a0298d5a47c0821c507f92524e70 (127.13.148.61:42183), 5e835ad474194259a0af1eab89906dbf (127.13.148.60:43929)
I20260926 22:53:10.374992 4200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d640c55f27c44c77b55d705ec7361d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e835ad474194259a0af1eab89906dbf"
I20260926 22:53:10.375103 4200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:10.373626 4246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.376303 4246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:10.376895 4200 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d640c55f27c44c77b55d705ec7361d96 in term 1.
I20260926 22:53:10.377108 4025 leader_election.cc:302] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e835ad474194259a0af1eab89906dbf, d640c55f27c44c77b55d705ec7361d96; no voters:
I20260926 22:53:10.377223 4246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 22:53:10.377375 4246 raft_consensus.cc:686] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [term 1 LEADER]: Becoming Leader. State: Replica: d640c55f27c44c77b55d705ec7361d96, State: Running, Role: LEADER
I20260926 22:53:10.377453 4246 consensus_queue.cc:237] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } }
I20260926 22:53:10.377916 4246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d640c55f27c44c77b55d705ec7361d96. Latest consensus state: current_term: 1 leader_uuid: "d640c55f27c44c77b55d705ec7361d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.378003 4246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:10.376158 4008 master_runner.cc:186] Error getting master registration for 127.13.148.61:42183: OK, instance_id { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" instance_seqno: 1790463190177753 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 22:53:10.378782 4008 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 22:53:10.379242 4257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:53:10.379398 4257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:53:10.379504 4267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 22:53:10.379848 4267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 22:53:10.386305 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.377399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.13.148.61:42183)
0926 22:53:10.377438 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.377440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.378664 (+ 1224us) server_negotiation.cc:407] Connection header received
0926 22:53:10.378717 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.378719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.378731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.378793 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.379522 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.380293 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.381322 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.382112 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.382154 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.383601 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.383603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.383603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.383610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.385102 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.385464 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.385465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.385466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.385517 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.385710 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.385710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.385711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.385860 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.385928 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.386132 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.386215 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 22:53:10.386853 4200 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [term 1 FOLLOWER]: Refusing update from remote peer d640c55f27c44c77b55d705ec7361d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:53:10.387339 4262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:53:10.388592 4232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d640c55f27c44c77b55d705ec7361d96. Latest consensus state: current_term: 1 leader_uuid: "d640c55f27c44c77b55d705ec7361d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.388688 4232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:10.389581 4262 sys_catalog.cc:455] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d640c55f27c44c77b55d705ec7361d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.389668 4262 sys_catalog.cc:458] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: This master's current role is: LEADER
I20260926 22:53:10.389977 4232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d640c55f27c44c77b55d705ec7361d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.390056 4232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:10.390602 4270 mvcc.cc:205] Tried to move back new op lower bound from 7333737227820953600 to 7333737227786899456. Current Snapshot: MvccSnapshot[applied={T|T < 7333737227820953600}]
I20260926 22:53:10.391021 4267 catalog_manager.cc:1347] Generated new cluster ID: 3fc3ac5a3ce6471ebbe07c2809a48b4a
I20260926 22:53:10.391059 4267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 22:53:10.393056 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.377169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.13.148.62:36709)
0926 22:53:10.377206 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.377208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.381134 (+ 3926us) server_negotiation.cc:407] Connection header received
0926 22:53:10.381280 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.381281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.381291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.381335 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.382481 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.383172 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.384660 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.384949 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.384988 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.388051 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.388052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.388053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.388070 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.389525 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.391602 (+ 2077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.391604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.391604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.391678 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.392172 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.392173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.392173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.392316 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.392404 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.392894 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.392968 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 22:53:10.400748 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.377828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.13.148.60:43929)
0926 22:53:10.377860 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.377861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.385512 (+ 7651us) server_negotiation.cc:407] Connection header received
0926 22:53:10.386901 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.386902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.386924 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.386952 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.390497 (+ 3545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.391193 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.395017 (+ 3824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.395327 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.395365 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.397526 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.397527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.397527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.397534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.399063 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.399665 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.399680 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.399681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.399737 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.400039 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.400040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.400040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.400179 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.400263 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.400582 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.400657 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:53:10.405828 4267 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 22:53:10.406474 4267 catalog_manager.cc:1504] Loading token signing keys...
I20260926 22:53:10.408587 4151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } attempt: 3
Sep 26 22:53:10 dist-test-slave-98hb krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790463190, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.148.61@KRBTEST.COM for kudu/127.13.148.60@KRBTEST.COM
I20260926 22:53:10.416399 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.408829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.148.60:43929 (local address 127.0.0.1:51576)
0926 22:53:10.408873 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 22:53:10.408882 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 22:53:10.408915 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 22:53:10.409040 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 22:53:10.409040 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 22:53:10.409206 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 22:53:10.409259 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.409261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.410156 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.410156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.410889 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 22:53:10.410891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.411328 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.411328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 22:53:10.411379 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.411763 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 22:53:10.411767 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 22:53:10.413131 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 22:53:10.415082 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.415083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.415084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.415298 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.415595 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 22:53:10.415596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 22:53:10.415596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 22:53:10.415666 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 22:53:10.416122 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 22:53:10.416123 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 22:53:10.416166 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 22:53:10.416211 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 22:53:10.416300 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 22:53:10.417133 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:10.408861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.13.148.60:43929)
0926 22:53:10.408939 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.408940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.408948 (+ 8us) server_negotiation.cc:407] Connection header received
0926 22:53:10.408968 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.408969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.408978 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.409036 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.409312 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.410073 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.410970 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:10.411261 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:10.411329 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:10.413219 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 22:53:10.413221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 22:53:10.413221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 22:53:10.413228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 22:53:10.414964 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.415391 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.415392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.415424 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.415505 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 22:53:10.415746 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 22:53:10.415747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 22:53:10.415747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 22:53:10.415991 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 22:53:10.416792 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:10.416864 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 22:53:10.416991 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 22:53:10.418741 4151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: Bootstrap starting.
I20260926 22:53:10.419415 4151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: Neither blocks nor log segments found. Creating new log.
I20260926 22:53:10.419682 4151 log.cc:824] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: Log is configured to *not* fsync() on all Append() calls
I20260926 22:53:10.420598 4151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: No bootstrap required, opened a new log
I20260926 22:53:10.422389 4151 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } }
I20260926 22:53:10.422508 4151 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd76a0298d5a47c0821c507f92524e70, State: Initialized, Role: FOLLOWER
I20260926 22:53:10.422580 4151 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } }
I20260926 22:53:10.422863 4151 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 22:53:10.422926 4125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d640c55f27c44c77b55d705ec7361d96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd76a0298d5a47c0821c507f92524e70"
I20260926 22:53:10.423010 4124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 22:53:10.424089 4124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [term 1 FOLLOWER]: Refusing update from remote peer d640c55f27c44c77b55d705ec7361d96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 22:53:10.424408 4262 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 22:53:10.425307 4267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96: Generated new TSK 0
I20260926 22:53:10.425472 4267 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 22:53:10.425624 4104 tablet.cc:2369] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 22:53:10.425824 4125 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d640c55f27c44c77b55d705ec7361d96 in term 1.
I20260926 22:53:10.422876 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.427059 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:10.427330 4276 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d640c55f27c44c77b55d705ec7361d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.427392 4276 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:10.428000 4288 mvcc.cc:205] Tried to move back new op lower bound from 7333737227897487360 to 7333737227786899456. Current Snapshot: MvccSnapshot[applied={T|T < 7333737227820953600}]
W20260926 22:53:10.428416 4290 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 22:53:10.428457 4290 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 22:53:10.427544 4287 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d640c55f27c44c77b55d705ec7361d96. Latest consensus state: current_term: 1 leader_uuid: "d640c55f27c44c77b55d705ec7361d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.429934 4287 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 22:53:10.430547 4246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d640c55f27c44c77b55d705ec7361d96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d640c55f27c44c77b55d705ec7361d96" member_type: VOTER last_known_addr { host: "127.13.148.62" port: 36709 } } peers { permanent_uuid: "bd76a0298d5a47c0821c507f92524e70" member_type: VOTER last_known_addr { host: "127.13.148.61" port: 42183 } } peers { permanent_uuid: "5e835ad474194259a0af1eab89906dbf" member_type: VOTER last_known_addr { host: "127.13.148.60" port: 43929 } } }
I20260926 22:53:10.430629 4246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d640c55f27c44c77b55d705ec7361d96 [sys.catalog]: This master's current role is: LEADER
W20260926 22:53:10.454738 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.453084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.13.148.61:42183)
0926 22:53:10.453128 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.453130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.454080 (+ 950us) server_negotiation.cc:407] Connection header received
0926 22:53:10.454184 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.454185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.454196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.454225 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.454674 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33328: BlockingRecv error: recv got EOF from 127.0.0.1:33328 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 22:53:10.455458 4079 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.453481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.13.148.62:36709)
0926 22:53:10.453518 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.453520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.454975 (+ 1455us) server_negotiation.cc:407] Connection header received
0926 22:53:10.455067 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.455068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.455078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.455107 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.455400 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60152: BlockingRecv error: recv got EOF from 127.0.0.1:60152 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 22:53:10.456182 4230 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.453824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.13.148.60:43929)
0926 22:53:10.453858 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.453860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.455682 (+ 1822us) server_negotiation.cc:407] Connection header received
0926 22:53:10.455776 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.455777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.455786 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.455814 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.456128 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51590: BlockingRecv error: recv got EOF from 127.0.0.1:51590 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 22:53:10.459730 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.458964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33342 (local address 127.13.148.61:42183)
0926 22:53:10.458997 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.458999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.459214 (+ 215us) server_negotiation.cc:407] Connection header received
0926 22:53:10.459316 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.459317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.459335 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.459362 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.459686 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33342: BlockingRecv error: recv got EOF from 127.0.0.1:33342 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:53:10.462656 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.461916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.13.148.61:42183)
0926 22:53:10.461947 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.461949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.462185 (+ 236us) server_negotiation.cc:407] Connection header received
0926 22:53:10.462276 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.462276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.462286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.462313 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.462604 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33358: BlockingRecv error: recv got EOF from 127.0.0.1:33358 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:53:10.486400 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.477877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.13.148.61:42183)
0926 22:53:10.477912 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.477914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.485889 (+ 7975us) server_negotiation.cc:407] Connection header received
0926 22:53:10.485961 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.485962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.485974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.486032 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.486345 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33370: BlockingRecv error: recv got EOF from 127.0.0.1:33370 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 22:53:10.494890 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.494069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.13.148.61:42183)
0926 22:53:10.494104 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.494106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.494353 (+ 247us) server_negotiation.cc:407] Connection header received
0926 22:53:10.494447 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.494449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.494458 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.494485 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.494839 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33374: BlockingRecv error: recv got EOF from 127.0.0.1:33374 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 22:53:10.504901 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.504036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.13.148.61:42183)
0926 22:53:10.504085 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.504086 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.504344 (+ 258us) server_negotiation.cc:407] Connection header received
0926 22:53:10.504443 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.504444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.504455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.504483 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.504836 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33378: BlockingRecv error: recv got EOF from 127.0.0.1:33378 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 22:53:10.522578 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.521981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.13.148.61:42183)
0926 22:53:10.522020 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.522022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.522129 (+ 107us) server_negotiation.cc:407] Connection header received
0926 22:53:10.522190 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.522191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.522201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.522247 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.522525 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33384: BlockingRecv error: recv got EOF from 127.0.0.1:33384 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 22:53:10.556209 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.555547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.13.148.61:42183)
0926 22:53:10.555592 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.555594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.555711 (+ 117us) server_negotiation.cc:407] Connection header received
0926 22:53:10.555774 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.555775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.555786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.555829 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.556154 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33388: BlockingRecv error: recv got EOF from 127.0.0.1:33388 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 22:53:10.622095 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.621410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.13.148.61:42183)
0926 22:53:10.621462 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.621464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.621552 (+ 88us) server_negotiation.cc:407] Connection header received
0926 22:53:10.621621 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.621622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.621631 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.621672 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.622032 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33394: BlockingRecv error: recv got EOF from 127.0.0.1:33394 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 22:53:10.751782 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:10.751144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.13.148.61:42183)
0926 22:53:10.751223 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:10.751225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:10.751287 (+ 62us) server_negotiation.cc:407] Connection header received
0926 22:53:10.751355 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:10.751356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:10.751367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:10.751415 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:10.751722 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33406: BlockingRecv error: recv got EOF from 127.0.0.1:33406 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 22:53:11.009411 4155 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 22:53:11.008816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.13.148.61:42183)
0926 22:53:11.008880 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:11.008882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:11.008909 (+ 27us) server_negotiation.cc:407] Connection header received
0926 22:53:11.008971 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:11.008972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:11.008983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:11.009033 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 22:53:11.009351 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33416: BlockingRecv error: recv got EOF from 127.0.0.1:33416 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 22:53:11.350569 4243 catalog_manager.cc:1259] Loaded cluster ID: 3fc3ac5a3ce6471ebbe07c2809a48b4a
I20260926 22:53:11.350629 4243 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf: loading cluster ID for follower catalog manager: success
I20260926 22:53:11.351408 4243 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf: acquiring CA information for follower catalog manager: success
I20260926 22:53:11.351692 4243 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e835ad474194259a0af1eab89906dbf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:53:11.428848 4290 catalog_manager.cc:1259] Loaded cluster ID: 3fc3ac5a3ce6471ebbe07c2809a48b4a
I20260926 22:53:11.428921 4290 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: loading cluster ID for follower catalog manager: success
I20260926 22:53:11.429713 4290 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: acquiring CA information for follower catalog manager: success
I20260926 22:53:11.430037 4290 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd76a0298d5a47c0821c507f92524e70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 22:53:11.524811 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:11.522479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.13.148.61:42183)
0926 22:53:11.522580 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:11.522582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:11.522592 (+ 10us) server_negotiation.cc:407] Connection header received
0926 22:53:11.522639 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:11.522640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:11.522686 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:11.522752 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:11.522859 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.523405 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.524195 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.524389 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.524440 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:11.524545 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:11.524654 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:11.524706 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:11.524719 (+ 13us) server_negotiation.cc:299] Negotiation successful
0926 22:53:11.524733 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
I20260926 22:53:11.528503 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:11.526526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.13.148.60:43929)
0926 22:53:11.526624 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:11.526626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:11.526633 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:53:11.526668 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:11.526668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:11.526700 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:11.526762 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:11.526841 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.527306 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.527941 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.528115 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.528164 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:11.528245 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:11.528337 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:11.528384 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:11.528410 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 22:53:11.528426 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260926 22:53:11.532481 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:11.529947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.13.148.61:42183)
0926 22:53:11.530040 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:11.530042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:11.530198 (+ 156us) server_negotiation.cc:407] Connection header received
0926 22:53:11.530289 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:11.530290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:11.530301 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:11.530352 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:11.530444 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.530930 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.531793 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.532050 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.532102 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:11.532250 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:11.532281 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:11.532305 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:11.532399 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 22:53:11.532423 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 22:53:11.532814 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:11.530142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.13.148.60:43929)
0926 22:53:11.530179 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:11.530181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:11.530188 (+ 7us) server_negotiation.cc:407] Connection header received
0926 22:53:11.530279 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:11.530280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:11.530290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:11.530316 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:11.530475 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.531166 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.531950 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.532223 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.532284 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:11.532580 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:11.532612 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:11.532654 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:11.532731 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 22:53:11.532754 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 22:53:11.533170 4371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 22:53:11.529953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.13.148.62:36709)
0926 22:53:11.530050 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 22:53:11.530052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 22:53:11.530147 (+ 95us) server_negotiation.cc:407] Connection header received
0926 22:53:11.530267 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 22:53:11.530269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 22:53:11.530319 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 22:53:11.530388 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 22:53:11.530546 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.531277 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.532189 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 22:53:11.532470 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 22:53:11.532529 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 22:53:11.532728 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 22:53:11.532840 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 22:53:11.532876 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 22:53:11.533004 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 22:53:11.533028 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":47,"threads_started":1}
I20260926 22:53:11.534562 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 4008
I20260926 22:53:11.539330 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 4083
I20260926 22:53:11.543264 13904 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskClAdwy/build/release/bin/kudu with pid 4158
2026-09-26T22:53:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2685 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5372 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (256135 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST